Fixing "0/N Nodes Are Available" for a GPU Pod

The scheduler tells you exactly why it failed, per node, with counts. Most time lost to this error is lost by treating the message as a generic complaint instead of as the itemised answer it is.

Reading the message

The message arrives as a FailedScheduling event on the Pending pod, visible with kubectl describe pod <name>. A typical one on a recent Kubernetes looks like this:

Warning  FailedScheduling  pod/inference-7c9d4  0/4 nodes are available:
  1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: },
  3 Insufficient nvidia.com/gpu.
  preemption: 0/4 nodes are available:
  4 No preemption victims found for incoming pod.

Parse it as a list of disjoint groups. Four nodes were considered; one was excluded for a taint, three for insufficient GPU. The counts must add up to the total, so if they do not, you are looking at a truncated message and should read the full event with kubectl get events --field-selector involvedObject.name=<pod>.

The word to be careful with is Insufficient. It does not mean the GPUs are busy in any hardware sense. It means: for this node, allocatable minus the sum of requests from pods already assigned is less than what this pod asked for. There are exactly two ways to get there — allocatable is too small, or the requests already booked are too large — and they have different fixes.

Cause 1: nothing is advertising a GPU

The most common cause on a new cluster, and the one that produces Insufficient nvidia.com/gpu on every node at once. Check allocatable:

kubectl get nodes -o custom-columns=\
NAME:.metadata.name,\
GPU:.status.allocatable.nvidia\.com/gpu

A column of <none> means the device plugin is not advertising. Within that, three sub-causes, in the order worth checking: the plugin DaemonSet is not running on those nodes; the plugin is running but the NVIDIA container runtime is not the node’s default low-level runtime, so the plugin’s own container sees no devices; or the driver is not loaded on the host at all. The plugin’s logs distinguish all three, and the sequence is laid out on installing the NVIDIA device plugin. Note that a healthy-looking plugin pod is not evidence of anything — it will report Ready while advertising zero.

Cause 2: every GPU is already allocated

If allocatable is non-zero, the GPUs are spoken for. Find out by whom:

kubectl describe node gpu-node-1 | grep -A6 "Allocated resources"

The requested count for nvidia.com/gpu will equal allocatable. Now find the holders across the cluster:

kubectl get pods -A -o json | jq -r '
  .items[]
  | select(any(.spec.containers[];
      .resources.limits["nvidia.com/gpu"] != null))
  | "\(.metadata.namespace)/\(.metadata.name) \(.spec.nodeName) \(.status.phase)"'

Look for pods in a terminal-ish state still holding a reservation: a Completed Job whose pod was never cleaned up still counts against the node until it is deleted, as does a pod stuck Terminating on a node that has stopped responding. Both are extremely common and both look like a capacity shortage. Set ttlSecondsAfterFinished on Jobs to stop the first from recurring.

If the reservations are legitimate, this is a capacity problem, and the fix is more nodes — see the cluster autoscaler on GPU node pools.

Cause 3: an untolerated taint

A clause of the form node(s) had untolerated taint {nvidia.com/gpu: present} is unambiguous: the node repels this pod. Read the node’s taints with kubectl get node gpu-node-1 -o jsonpath={.spec.taints} and add a matching toleration, as on taints and tolerations for a GPU node pool.

Two variants of this cause are less obvious. A taint added by the node itself — node.kubernetes.io/disk-pressure, node.kubernetes.io/not-ready, node.kubernetes.io/unreachable — is the node reporting a problem, and tolerating it is the wrong response; disk pressure on a GPU node is usually a root volume filled by large container images. And on a cluster relying on the ExtendedResourceToleration admission plugin to inject tolerations automatically, a pod that requests no extended resource gets no toleration, which is why a sidecar-only pod fails where the model pod succeeds.

Cause 4: the resource name does not exist

Insufficient nvidia.com/gpu on a node that clearly has GPUs is usually a naming mismatch, and there are three ways to produce one.

• Time-slicing with renaming on. With renameByDefault: true the node advertises nvidia.com/gpu.shared and nothing at all under nvidia.com/gpu. Every existing manifest breaks at once. See GPU time-slicing.
• MIG in mixed strategy. Partitions are advertised as nvidia.com/mig-1g.10gb and similar; a request for nvidia.com/gpu matches nothing on a fully partitioned node. See MIG partitioning.
• A typo. Extended resource names are not validated against anything, so nvidia.com/gpus is a perfectly legal request for a resource no node has. Compare the pod’s request string against the node’s allocatable keys character by character rather than by eye.

Causes 5 and 6: quota and capacity

A namespace ResourceQuota does not produce this message at all, which is worth knowing because it saves you looking. Quota is enforced at admission, so the pod is rejected before it ever becomes Pending, with a message of the form:

Error from server (Forbidden): pods "inference-1" is forbidden:
exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=2,
used: requests.nvidia.com/gpu=3, limited: requests.nvidia.com/gpu=4

If the pod was created by a Deployment you will not see that on the pod, because no pod exists — it appears on the ReplicaSet, via kubectl describe rs <name>. A Deployment stuck below its desired replica count with no Pending pods anywhere is almost always this. Note also that Kubernetes only permits the requests. prefix for extended resources in a quota, because they cannot be overcommitted, so limits.nvidia.com/gpu in a quota spec is not the thing you want. Namespace GPU quota covers it properly.

The last cause is not in the cluster. If nodes should be appearing and are not, the autoscaler may be requesting instances the cloud provider cannot supply, or the account may be at an accelerator quota limit. Read the autoscaler status ConfigMap and the provider’s quota console for the exact limit name and current value; those are figures to look up rather than assume, and they are the one part of this diagnosis that Kubernetes cannot answer for you.