Scheduling GPU Pods on Kubernetes

Scheduling a GPU pod is one line in a pod spec. The work is in confirming that the line did what you think it did, because a GPU pod that runs on a node with no GPU visible to it fails at import time, not at scheduling time.

What the scheduler is matching

The scheduler has no concept of a GPU. It has a concept of an extended resource: a named, integer-valued quantity that a node advertises in status.allocatable, and that a pod consumes by naming it in resources.limits. The NVIDIA device plugin is what puts nvidia.com/gpu in that map. Without the plugin running, the node has GPUs, nvidia-smi on the host sees them, and Kubernetes advertises none.

So before writing any pod, read the number the scheduler will read:

kubectl get nodes -o custom-columns=\
NAME:.metadata.name,\
GPU:.status.allocatable.nvidia\.com/gpu

A column of <none> means the device plugin is not advertising, and no pod spec will fix that — start at installing the device plugin instead. A number is the ceiling on how many single-GPU pods can be simultaneously running on that node, forever, regardless of how idle the hardware looks. The scheduler subtracts a pod’s request from allocatable the moment it binds, and adds it back when the pod terminates. Utilisation never enters the calculation.

Filtering, then scoring

The scheduler runs two phases over the node list, and the GPU participates in only one of them. Filtering removes nodes the pod cannot run on — the NodeResourcesFit plugin is what checks that allocatable minus existing requests still covers this pod’s nvidia.com/gpu, alongside taints, affinities and node selectors. Scoring then ranks whatever survived, and the default scoring strategy for resources is LeastAllocated: it prefers the emptiest node.

That default is deliberate for stateless services and awkward for accelerators. Spreading four single-GPU pods across four eight-GPU nodes is the textbook result of LeastAllocated, and it leaves you paying for four nodes to do work that fits on one, with no node idle enough for an autoscaler to remove. Changing it means a scheduler configuration that scores with MostAllocated, optionally weighted so that nvidia.com/gpu dominates the score — which is a cluster-level decision rather than something a pod spec can express, and is covered on bin-packing GPU pods.

Two consequences are worth holding on to. Scoring never looks at utilisation, only at requests, so a node whose GPUs are allocated but idle scores exactly as full as one running flat out. And the scheduler picks the node, not the device: which physical GPU a container receives is decided by the kubelet when it asks the device plugin to allocate, which is why you cannot pin a pod to GPU 3 through any field in a pod spec.

The pod spec

The GPU goes in limits. You may also write the same value in requests, and Kubernetes documents that the two must then be equal; you may not write requests alone. Omitting requests is the normal form, because the limit is copied into the request automatically.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-probe
spec:
  restartPolicy: Never
  containers:
    - name: probe
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi", "-L"]
      resources:
        limits:
          nvidia.com/gpu: 1

Two details in that manifest are load-bearing. restartPolicy: Never keeps a one-shot probe from looping, and the image matters: a plain ubuntu image will schedule fine and then have no nvidia-smi to run, which reads like a GPU problem and is not one. The CUDA base image is the smallest thing that proves the device was injected.

Confirming the placement

1. Apply it and watch where it lands: kubectl get pod gpu-probe -o wide. The NODE column is the answer to “did the scheduler place it on a GPU node”.
2. Read the logs: kubectl logs gpu-probe. You want a line beginning GPU 0: NVIDIA followed by a UUID. That output comes from inside the container, so it proves the device node was actually mounted rather than merely accounted for.
3. Confirm the accounting on the node: kubectl describe node <node> and look at the Allocated resources table. nvidia.com/gpu should show 1 requested and 1 limit while the pod runs.
4. If the pod is Pending instead, kubectl describe pod gpu-probe prints a FailedScheduling event whose message names every reason per node count. That message is the whole diagnosis and it is covered on the insufficient-GPU fix page.

Pinning to a particular GPU model

nvidia.com/gpu: 1 asks for “a GPU”, and in a cluster with mixed hardware that can put a 70B model on the wrong card. NVIDIA’s GPU Feature Discovery component labels nodes with hardware facts — product name, memory, count, driver version — and those labels are what a nodeSelector or a nodeAffinity block matches on.

  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB

Do not type that value from memory. Read the labels your cluster actually carries with kubectl get node <node> -o jsonpath={.metadata.labels} and copy the string, because the product label is generated from what the driver reports and differs between SXM and PCIe variants of the same card. An affinity expression with In over several products is usually better than an exact selector: it lets the pod run on anything big enough rather than waiting for one model. Node affinity by GPU type goes into the operator forms.

The three things that go wrong first

• The pod schedules onto a CPU node. This means the GPU line is not being enforced — usually a typo in the resource name. nvidia.com/gpus or nvidia/gpu is not an error; it is an unknown extended resource that no node advertises, so the pod stays Pending, or worse, a mistyped key nested under the wrong field is silently dropped and the pod schedules anywhere.
• The pod runs but sees no device. Almost always the container runtime: the NVIDIA runtime has to be wired in for the device nodes and driver libraries to be injected into the container. The pod is scheduled correctly and the workload still cannot see a GPU.
• The second replica goes Pending. Expected, and not a bug. One physical GPU is one allocatable unit. Getting more pods than GPUs requires time-slicing or MIG partitioning, and both change what the node advertises rather than what the pod asks for.

Once a single-GPU probe places and prints its UUID, the same resources.limits block is all a real serving Deployment needs. Everything after that — which node pool, how many replicas, when the node appears — is scheduling policy layered on top of a request that already works.