Taints and Tolerations for a GPU Node Pool

A GPU node is the most expensive node in the cluster and, to the scheduler, the most attractive one: it usually has the most CPU and the most memory free. Without a taint it will quietly fill with logging sidecars and batch jobs.

Why an untainted GPU pool fills with the wrong pods

The default scheduler spreads pods across nodes with room. A GPU instance type is generally paired with a large CPU and memory allocation, so it presents as the emptiest node in the cluster and wins scoring for any pod that does not care where it goes. Those pods do not consume nvidia.com/gpu, so GPU scheduling still works — right up to the moment the node needs to drain or scale down, which it now cannot, because something unrelated is running on it.

That is the real cost, and it is not primarily about performance. A GPU node that cannot be removed because a CPU workload sits on it is the most expensive idle resource you own. Autoscalers respect PodDisruptionBudgets and non-evictable pods, so one stray pod pins an accelerator instance indefinitely.

Applying the taint

The convention is nvidia.com/gpu=present:NoSchedule, which is worth adopting even on clusters where you would prefer your own key, because tooling recognises it. Google documents that GKE applies exactly this taint automatically when you add a GPU node pool to a cluster that already has a non-GPU pool.

kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule

Do not do it that way for anything permanent. A node created later by an autoscaler will not carry a taint applied by kubectl, so the taint has to belong to the node pool definition:

• On a managed node group, the taint is a field on the group and is re-applied to every node it launches.
• On Karpenter it is spec.template.spec.taints on the NodePool, as in scaling GPU nodes to zero with Karpenter.
• On the kubelet directly it is --register-with-taints, which applies at node registration and so covers the window before any controller reconciles.

The three effects differ in what they do to pods that are already there. NoSchedule blocks new placements and leaves running pods alone. PreferNoSchedule is a scoring penalty rather than a rule, which for a GPU pool means it will eventually be violated under pressure. NoExecute evicts pods that do not tolerate it, and adding it to a live pool will evict your monitoring agents. Start with NoSchedule.

The matching toleration

A toleration matches on key, operator, value and effect. For a taint whose value carries no information, Exists is the form to use — it keeps working if somebody changes present to true:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference
spec:
  replicas: 2
  selector:
    matchLabels: { app: inference }
  template:
    metadata:
      labels: { app: inference }
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      nodeSelector:
        nvidia.com/gpu.present: "true"
      containers:
        - name: server
          image: registry.example/inference:1.4.0
          resources:
            limits:
              nvidia.com/gpu: 1

You may not have to write the toleration at all. Kubernetes ships an admission plugin, ExtendedResourceToleration, which adds a toleration automatically to any pod that requests an extended resource, keyed on the resource name. Google documents GKE as running it, which is why GPU pods schedule onto GKE’s auto-tainted pools without a toleration in the manifest. On a cluster where it is not enabled, the same manifest is Pending. Writing the toleration explicitly costs six lines and works in both cases.

A toleration is not an attraction

This is the point the taint documentation states once and everyone forgets. A taint repels; a toleration removes the repulsion. Neither expresses a preference. A pod with a GPU toleration and no other constraint is free to run anywhere in the cluster, including on the CPU nodes.

For a pod requesting nvidia.com/gpu this is harmless, because the resource request itself is the constraint that forces a GPU node. It bites on the pods around the model server — a sidecar, a metrics scraper, a data loader — that were given the toleration “so they can run with the model” and now roam freely. If a pod must be on a GPU node, it needs a nodeSelector or a nodeAffinity as well; the toleration only makes that placement legal. See node affinity by GPU type for the selection half.

What a taint does not do to running pods

Tainting a node that is already full does not clean it up. With NoSchedule, existing pods stay exactly where they are, forever, because the effect is evaluated at binding time and never re-evaluated. The usual surprise is applying the taint to fix a crowded GPU pool and finding the pool just as crowded an hour later.

The two ways to actually remove them differ in blast radius. kubectl drain cordons the node and evicts its pods through the eviction API, which respects PodDisruptionBudgets and gives each container its termination grace period — the correct tool, and the one to reach for on a GPU node where a running generation should be allowed to finish. Adding a NoExecute taint instead evicts everything that does not tolerate it, immediately and without regard to budgets, which on a mixed node means your monitoring agents go too. Cordoning is itself a taint, incidentally: node.kubernetes.io/unschedulable with effect NoSchedule, which is why a cordoned node behaves identically to a tainted one.

A toleration for a NoExecute taint may also carry tolerationSeconds, which means “tolerate this for a while, then leave”. Kubernetes uses it by default for the node-condition taints, so a pod on an unreachable node is evicted after a delay rather than instantly. For a GPU workload that delay is worth lengthening: moving a model server costs minutes of reload, and a node that is briefly unreachable is often back before a replacement would have finished loading.

Taints also feed the autoscalers, which is a reason to keep them declarative. Both the cluster autoscaler and Karpenter simulate whether a Pending pod could run on a node from a given pool, and that simulation uses the pool’s declared taints. A taint applied by hand to running nodes but absent from the pool definition makes the simulation optimistic: new nodes get provisioned for pods that will then be repelled by them. The cluster autoscaler page gives the tag that declares a taint for a node group sitting at zero.

The pods that must also tolerate it

Tainting a GPU pool excludes cluster infrastructure too, and the failure is worse than it looks:

• The NVIDIA device plugin. If it cannot land on the tainted node, the node never advertises nvidia.com/gpu, so no GPU pod can schedule anywhere. NVIDIA’s manifests carry tolerations for the standard key; a custom taint key needs adding to the DaemonSet or the chart values.
• CNI and kube-proxy DaemonSets. Usually shipped with a blanket toleration, but worth confirming on a cluster with an unusual network plugin — a node without CNI stays NotReady and looks like a hardware fault.
• Monitoring and log agents. A GPU node with no node-exporter and no DCGM exporter is invisible exactly where visibility is most expensive. GPU monitoring covers what those agents should be collecting.

Verify from the node rather than from the manifests. After tainting, kubectl get pods --all-namespaces --field-selector spec.nodeName=gpu-node-1 should list your GPU workloads plus the DaemonSets you expect and nothing else. Anything missing from that list is something you have stopped monitoring or stopped networking.