Node Affinity for Pinning Inference Pods to a GPU Type

nvidia.com/gpu: 1 asks for a GPU. It does not ask for a particular one, and on a mixed cluster the scheduler is free to put a 70B model on the card that cannot hold it. Node affinity is how you say which.

Why a GPU request is not enough

The device plugin advertises a single opaque resource name. Every GPU in the cluster, whatever it is, adds one to a node’s nvidia.com/gpu capacity, and the scheduler treats those units as interchangeable because from its perspective they are. It has no model of memory per device, no model of compute capability, and no way to know that your pod needs 80 GB rather than 16.

On a homogeneous cluster this is fine and you should not add affinity at all — a redundant constraint is a future outage. It stops being fine the moment a second GPU type appears, which happens sooner than people expect: a spot pool of a cheaper card, a new node group after a quota increase, a region where the original instance type is unavailable. After that, a pod without affinity is a coin flip, and the failure mode is a CUDA out-of-memory error at model load that looks like a configuration bug rather than a scheduling one.

Memory is the obvious dimension and it is not the only one. Compute capability decides whether a kernel compiled for a newer architecture will run at all — an image built assuming one generation fails on an older card with a CUDA error about no kernel image being available, which reads like a broken build rather than a misplacement. Interconnect matters too: a multi-GPU pod using tensor parallelism across cards linked by NVLink behaves very differently from the same pod on cards talking over PCIe, and nothing in the manifest distinguishes them. Neither of those is visible to the scheduler either, and both are reasons to pin the SKU rather than trust the resource count.

Which label to match on

NVIDIA’s GPU Feature Discovery generates node labels describing the devices present, and it is a separate component from the device plugin — installing the plugin alone gives you the resource and none of the labels. Its documentation describes it as automatically generating labels for the set of GPUs available on a node, using Node Feature Discovery to apply them. The label you almost always want is nvidia.com/gpu.product, whose value is a sanitised device name such as NVIDIA-H100-80GB-HBM3. Companions include nvidia.com/gpu.count and nvidia.com/gpu.memory.

Managed platforms also apply their own accelerator labels, and cloud instance types are exposed through the standard node.kubernetes.io/instance-type key. Any of the three can work. The argument for the NVIDIA label is that it describes the hardware rather than the SKU that happens to contain it, so it survives a change of instance family; the argument for the instance-type label is that it exists without installing anything. Pick one convention per cluster and write it down, because a fleet with two conventions is a fleet where half the manifests silently match nothing.

Required affinity: it must be this GPU

requiredDuringSchedulingIgnoredDuringExecution is a hard filter. A node that does not match is eliminated, and if none match the pod stays Pending indefinitely — which is the correct outcome when the alternative is an out-of-memory crash loop, and the wrong one when you would have accepted a slower card.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-70b
spec:
  replicas: 2
  selector:
    matchLabels: { app: llama-70b }
  template:
    metadata:
      labels: { app: llama-70b }
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nvidia.com/gpu.product
                    operator: In
                    values:
                      - NVIDIA-H100-80GB-HBM3
                      - NVIDIA-A100-SXM4-80GB
      containers:
        - name: server
          image: registry.example.com/inference:1.9.0
          resources:
            limits:
              nvidia.com/gpu: "1"

Two details in that block are load-bearing. Multiple entries under one matchExpressions are ANDed; multiple nodeSelectorTerms are ORed. And operator: In with a list of values is itself an OR — the manifest above says “an 80 GB H100 or an 80 GB A100”, which is usually what “a card big enough” means in practice. Expressing the requirement as a set rather than a single string is the difference between a resilient deployment and one that goes Pending the day you add a node group.

The IgnoredDuringExecution half of the name is a promise: a running pod is not evicted if the node’s labels later stop matching. Affinity constrains placement only.

The obvious question is why not use nodeSelector, which expresses the same idea in three lines. Two reasons. nodeSelector supports only exact equality, so it cannot say “one of these two cards” — and a single-value constraint is precisely the version that strands your pods when the fleet changes. And it has no preferred form, so there is no path from “must” to “would rather” without rewriting the spec. Affinity also offers NotIn, Exists, DoesNotExist, Gt and Lt; NotIn in particular is the clean way to say “anything except the old generation” without enumerating every card you do want.

One limitation catches people who reach for Gt on GPU memory. The numeric operators compare integer label values, and NVIDIA’s nvidia.com/gpu.memory is such a label, so a rule like “memory greater than 40000” is expressible and is often a better constraint than a list of product names — it keeps working when a new card arrives. It is still string-typed underneath, so a label value with a unit suffix will not compare; check what your nodes actually carry before depending on it.

Preferred affinity: rank, do not exclude

preferredDuringSchedulingIgnoredDuringExecution contributes to a node’s score instead of filtering it. Every matching node gets its weight added; nothing is excluded. Use it when a fallback is genuinely acceptable.

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nvidia.com/gpu.present
                    operator: Exists
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: nvidia.com/gpu.product
                    operator: In
                    values: ["NVIDIA-H100-80GB-HBM3"]
            - weight: 20
              preference:
                matchExpressions:
                  - key: nvidia.com/gpu.product
                    operator: In
                    values: ["NVIDIA-L40S"]

Weights run from 1 to 100 and are summed across all matching preferences, then combined with every other scoring plugin. That last clause is the one that surprises people: a preference of weight 100 is not a guarantee, because balanced-allocation and image-locality scores are being added at the same time and can outvote it. If the outcome must be certain, the rule belongs in required.

Verifying, and the failure to expect

1. Confirm the labels exist before you rely on them: kubectl get nodes -L nvidia.com/gpu.product. An empty column means GPU Feature Discovery is not running, and every required rule you write will match nothing.
2. Apply, then check placement rather than status: kubectl get pods -o wide shows the node each replica landed on. Cross-check that node’s product label.
3. Deliberately break it once. Change one character of the value and re-apply; the pod should go Pending with a FailedScheduling clause counting every node under “didn’t match Pod’s node affinity/selector”. Knowing what your own mistake looks like is worth the two minutes — the full diagnosis is in fixing an unschedulable GPU pod.
4. If the target nodes are tainted — spot pools and dedicated GPU pools usually are — affinity alone still will not place the pod. Affinity attracts; a toleration is what permits. Taints and tolerations for GPU nodes covers the other half.