Fixing PodUnschedulable Because No Node Matches the GPU Selector

The pod is Pending. kubectl describe pod shows a FailedScheduling event and a line that begins “0/6 nodes are available”. That line contains the answer, and it is a more precise statement than it looks.

The message, read properly

The literal text is a FailedScheduling event on the pod, and the pod also carries a condition: type: PodScheduled, status: "False", reason: Unschedulable. That condition reason is where the word “unschedulable” comes from in most tooling; managed platforms and the cluster autoscaler surface it under names like PodUnschedulable in their own event streams. The useful text is the event message.

kubectl describe pod inference-7d9f8c-abcde -n inference

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/6 nodes are available:
    4 node(s) didn't match Pod's node affinity/selector,
    2 Insufficient nvidia.com/gpu.
    preemption: 0/6 nodes are available:
    4 Preemption is not helpful for scheduling,
    2 No preemption victims found for incoming pod.

Two clauses, two different problems, and they need different fixes. Four nodes were eliminated before resources were even considered, because the pod’s nodeSelector or required node affinity did not match their labels. The other two matched the labels and were rejected for capacity — that is the failure covered in Insufficient nvidia.com/gpu, and it is not this page.

Every node is in exactly one clause

The counts sum to the total. That is the property that makes the message diagnostic rather than decorative: the scheduler runs filter plugins over every node, and each rejected node is attributed to the first predicate that rejected it. If the numbers in your message do not add up to the node count, you are reading a truncated message — the scheduler summarises when there are many nodes — and you should look at the pod condition or raise the scheduler’s verbosity rather than reason from the excerpt.

The ordering matters too. Node selector and affinity are cheap filters and run early, so a pod that fails on labels never gets far enough to report a GPU shortage. A message that says only “didn’t match Pod’s node affinity/selector” for every node tells you nothing whatsoever about whether you have enough GPUs; you may have plenty, sitting on nodes the pod refused to consider.

The preemption: block is a second, independent report. “ Preemption is not helpful” against the affinity clause is expected: evicting a pod does not change a node’s labels, so there is nothing preemption could do. Ignore that block entirely while you are chasing a selector mismatch.

One more property of the message repays attention: it describes the last scheduling attempt, not a standing truth. Events expire — the default retention is an hour — so a pod that has been Pending since yesterday may have no FailedScheduling event at all, and kubectl describe pod will show an empty Events section that reads like the scheduler never looked at it. Reach for kubectl get events --field-selector involvedObject.name=POD -n NS first, and if that is empty too, force a fresh attempt by deleting the pod and letting its controller recreate it. A Pending pod with no explanation is almost always an expired event rather than a mystery.

Compare the selector against real labels

Two commands. Print what the pod asks for, then print what the nodes have, and diff them by eye.

# What the pod requires.
kubectl get pod inference-7d9f8c-abcde -n inference -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl get pod inference-7d9f8c-abcde -n inference \
  -o jsonpath='{.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution}{"\n"}'

# What the GPU nodes actually carry.
kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count,node.kubernetes.io/instance-type

# Or, exhaustively, for one node.
kubectl get node gpu-node-3 -o jsonpath='{.metadata.labels}{"\n"}' | tr ',' '\n'

The -L flag adds a label as a column, which is the fastest way to see whether a label exists at all across the fleet. An empty column means no node carries it — which is a different problem from carrying it with a different value, and the two have different fixes.

The four causes, in order of likelihood

• The label is not there because nothing installs it. Labels like nvidia.com/gpu.product are written by NVIDIA’s GPU Feature Discovery, which is a separate component from the device plugin that advertises the nvidia.com/gpu resource. A cluster can therefore schedule GPU pods perfectly well and still have no gpu.product label anywhere, because only the plugin was installed. See the NVIDIA device plugin.
• The value is right but not exact. Selector matching is byte-for-byte string equality. NVIDIA’s product labels use a sanitised form of the marketing name — the label value for an 80 GB H100 SXM part is not the string in the datasheet, and neither the case nor the hyphens are negotiable. Copy the value out of the node, never out of a document.
• You used the wrong vocabulary for the platform. Managed platforms add their own accelerator labels alongside NVIDIA’s. A selector written against a cloud-specific accelerator key works on that platform and matches nothing after a migration, and vice versa. Whichever you choose, choose one.
• The node exists in the plan, not in the cluster. If a node pool is scaled to zero and the autoscaler cannot infer the labels a not-yet-created node would have, it will not scale up for your pod. Autoscalers handle this with per-node-group label hints; if yours is missing them, the pod waits forever next to an empty pool that would have fitted it.

A fifth cause is worth separating out because it produces the same clause for a different reason: the labels are correct and the nodes are not Ready. A node that is NotReady, cordoned, or still running its GPU driver installation is excluded before the selector is even evaluated, and depending on which filter rejected it, the summary can attribute it to the affinity clause. Check kubectl get nodes for readiness and for SchedulingDisabled before you start editing manifests — a forgotten cordon after last week’s maintenance is a disproportionately common cause of this exact message.

The related trap is the driver race on a freshly created GPU node. The node joins, labels itself with everything the kubelet knows, and only later — once the driver installer and device plugin have run — gains both nvidia.com/gpu capacity and the GPU Feature Discovery labels. For a few minutes a real GPU node genuinely does not match a GPU selector, so a pod created during that window reports this failure and then schedules on its own. If the message appears only right after a scale-up, wait a minute and look again before changing anything.

Fixing it

1. Establish which label vocabulary your cluster actually has: kubectl get nodes -o json | jq -r '.items[].metadata.labels | keys[]' | sort -u | grep -i gpu. The output is the complete set of GPU-related keys available to you.
2. If nvidia.com/gpu.product is absent and you want it, install GPU Feature Discovery — it is included in the NVIDIA GPU Operator and available standalone in the device plugin repository. Nothing else will create those labels.
3. Correct the selector to a value copied verbatim from a node. Apply, and confirm the event clause count changes: if the affinity clause drops to zero, the label problem is solved even if you now have a capacity problem instead.
4. If you only needed “a GPU” rather than a specific one, delete the selector and let the nvidia.com/gpu resource request do the filtering. A resource request is already a constraint; adding a redundant label selector on top of it is the most common way to create this failure in the first place.
5. If you genuinely need one GPU type, express it as node affinity rather than nodeSelector so you can add a preferred fallback — node affinity for a GPU type covers that shape.