MIG Partitioning for Kubernetes GPU Nodes

MIG splits one physical GPU into hardware-isolated instances with their own memory, cache and streaming multiprocessors. Kubernetes then treats each instance as a separate schedulable device — but what that device is called depends on a strategy you choose before any of it works.

What MIG divides

Multi-Instance GPU is a hardware feature of NVIDIA’s Ampere architecture and later — A100, H100 and their successors — not a software scheduler. A MIG instance gets a fixed slice of memory and a fixed slice of compute, and a workload in one instance cannot read, starve or crash a workload in another. That is the entire reason to prefer it over time-slicing, which offers density with no isolation at all.

Partitions come from a fixed catalogue of profiles named for their compute and memory share: on an 80GB A100 the profiles NVIDIA documents include 1g.10gb, 2g.20gb, 3g.40gb and 7g.80gb; on the 40GB part the same slots appear as 1g.5gb, 2g.10gb, 3g.20gb and 7g.40gb. The first number is compute slices out of seven, the second is memory. You cannot invent a profile, and not every combination of profiles can coexist on one card — the valid layouts are fixed by the hardware.

single or mixed, decided once

The device plugin’s migStrategy has three values, and it decides the resource name every pod in the cluster will use.

• none — MIG is ignored. The plugin advertises whole GPUs. This is the default and is correct on a cluster with no MIG-capable hardware.
• single — every GPU on the node is in MIG mode with identical partitions, and the plugin advertises them as nvidia.com/gpu. Pod specs do not change at all: they still ask for nvidia.com/gpu: 1 and receive one partition instead of one card. This is by far the easier migration, because no manifest in the cluster needs editing.
• mixed — partitions are advertised under their profile names, as nvidia.com/mig-1g.10gb, nvidia.com/mig-3g.40gb and so on, alongside nvidia.com/gpu for any card left whole. Pods choose their size. This is the powerful option and it means every GPU manifest you own must name a specific profile.

The trap in mixed is that a pod asking for nvidia.com/gpu: 1 on a fully partitioned node will never schedule, because that resource no longer exists there. The scheduler reports Insufficient nvidia.com/gpu on a node visibly full of idle GPU capacity, which is one of the causes catalogued on the insufficient-GPU fix page. Choose single unless you actually need heterogeneous partition sizes on one node.

Applying a MIG configuration

MIG mode is set on the GPU itself, not in Kubernetes. NVIDIA’s GPU Operator ships a MIG manager that does it from a node label, which is the path worth using because it also restarts the plugin and re-advertises afterwards.

1. Install the GPU Operator with the MIG strategy set, for example --set mig.strategy=single on the operator release. This is the decision from the previous section and it is cluster-wide.
2. Inspect the generated ConfigMap of available configurations. NVIDIA documents entries such as all-disabled, all-1g.10gb, all-3g.40gb and all-balanced, discovered from the hardware present.
3. Drain the node, or accept that the manager will do it for you. The documentation is explicit that no user workloads may be running on the GPUs being configured, and the manager terminates GPU pods to achieve that. In some cloud environments the node also needs a reboot.
4. Apply the label: kubectl label node gpu-node-1 nvidia.com/mig.config=all-1g.10gb --overwrite.
5. Watch the status label rather than guessing: kubectl get node gpu-node-1 -o jsonpath={.metadata.labels.nvidia\.com/mig\.config\.state}. It moves through pending to success, or to failed — and failed with an explanation is far more useful than a partition list you have to interpret.
6. Confirm the new allocatable map with kubectl describe node gpu-node-1. Under single you should see nvidia.com/gpu: 7 where the card previously advertised 1; under mixed you should see the profile names.

Requesting a partition from a pod

Under mixed, the request names the profile and the rest of the spec is unchanged:

    resources:
      limits:
        nvidia.com/mig-1g.10gb: 1

The same integer rules apply as to any other GPU resource: whole numbers only, request equal to limit, no overcommitment. A pod cannot ask for two partitions and get a coherent larger one — two 1g.10gb instances are two isolated devices with 10GB each, not a 20GB device. If a model does not fit in a partition, it needs a bigger profile or a whole card, and there is no way to express “a partition of at least this size” in a resource request. That constraint is the practical argument for keeping the number of distinct profiles in a cluster very small. The reasoning behind the integer rule is on resource requests and limits for a GPU inference pod.

What partitioning costs you

• Reconfiguration is disruptive. Changing a node’s MIG layout evicts its GPU pods and can require a reboot. This is not a knob to turn in response to load; it is closer to a property of the node pool, which argues for separate pools per layout rather than one pool you retune.
• Capacity becomes lumpy. Seven 1g instances cannot serve one job that needs a whole card. A cluster partitioned for small tenants has no room for a large one, even when every partition is idle, and the autoscaler cannot help because the resource being requested does not exist anywhere.
• Quota needs to name every profile. A ResourceQuota limiting requests.nvidia.com/gpu constrains nothing about requests.nvidia.com/mig-1g.10gb. Each advertised resource is a separate quota key. Namespace GPU quota covers the syntax.
• Not every framework is happy. Multi-GPU collective operations across MIG instances are restricted, so a training job expecting NCCL across devices is a poor fit. MIG is aimed at many independent small workloads, which is exactly the shape of inference for small models.