Cluster Autoscaler and GPU Node Pools

The cluster autoscaler adds nodes when pods cannot be scheduled and removes them when they are underused. On GPU node groups both halves need configuring, and the scale-from-zero case needs configuring somewhere other than Kubernetes.

What the autoscaler does each interval

On every scan — documented as a 10-second default via --scan-interval — the autoscaler looks for pods the scheduler could not place, and for each candidate node group it simulates whether adding a node from that group would make them schedulable. If yes, it asks the cloud provider to grow the group. It does not create pods, does not talk to the HPA, and does not know anything about your workload beyond its resource requests, tolerations and affinities.

That simulation is the crux. It runs against a template of what a node in the group looks like, and for a group with at least one running node the template comes from that node. For an empty group there is no node to copy.

Scaling a GPU group from zero

The autoscaler has supported scaling to and from zero for a long time, but from zero it needs to be told what the nodes would offer. On AWS that is done with tags on the Auto Scaling group, following the documented k8s.io/cluster-autoscaler/node-template/ scheme:

k8s.io/cluster-autoscaler/enabled                                     = true
k8s.io/cluster-autoscaler/my-cluster                                 = owned
k8s.io/cluster-autoscaler/node-template/resources/nvidia.com/gpu     = 4
k8s.io/cluster-autoscaler/node-template/label/nvidia.com/gpu.present = true
k8s.io/cluster-autoscaler/node-template/taint/nvidia.com/gpu         = present:NoSchedule

Miss the resources tag and the simulation for an empty GPU group concludes the node would have no nvidia.com/gpu, so adding one would not help, so the group stays at zero forever. The pod sits Pending, the autoscaler logs that no group would help, and every node group looks correctly configured. Miss the taint tag and the opposite happens: the simulation thinks GPU pods can land on nodes that will actually repel them, and the group scales up into nodes that stay empty.

The label tag matters for the same reason whenever pods use a nodeSelector — a selector on a label the template does not declare makes the simulation fail. Every label and taint your GPU pods depend on has to appear here.

The flags that matter for GPUs

The documented defaults are tuned for stateless CPU workloads, and two of them behave differently on accelerators:

• --scale-down-unneeded-time, default 10m0s. How long a node must be unneeded before removal. On GPU nodes that take several minutes to become useful, ten minutes of idle is often too eager — but it is also ten minutes of paying for an idle accelerator, so the number is a direct cost decision rather than a safety one.
• --scale-down-utilization-threshold, default 0.5. The CPU/memory utilisation below which a node is a scale-down candidate.
• --scale-down-gpu-utilization-threshold, default 0.5. The GPU equivalent, and the important one: the FAQ states that utilisation calculation for an accelerator node only cares about the GPU resource, and CPU and memory utilisation are ignored. So a GPU node whose CPUs are busy is still a scale-down candidate if its GPUs are not allocated. Note that this is allocation, not activity — requests divided by allocatable, not anything read from the hardware.
• --scale-down-delay-after-add, default 10m0s. No scale-down for this long after a scale-up, which damps thrash during a burst.
• --max-node-provision-time, default 15m0s. How long the autoscaler waits for a requested node before giving up and considering the request failed. GPU instances can genuinely take a while, and a group that repeatedly hits this limit is a capacity problem, not a configuration one.

Where the scale-up delay comes from

“The autoscaler is slow” is usually a sum of stages, most of which are not the autoscaler. The pod must first fail to schedule, which the scheduler decides. The autoscaler notices on its next scan. The cloud provider allocates and boots an instance. The kubelet registers and the node goes Ready. The device plugin DaemonSet must land and advertise nvidia.com/gpu. Only then can the pod bind, and only then does it start pulling a multi-gigabyte image and loading weights.

The autoscaler’s own contribution is a scan interval. Everything else is boot, plugin and image, which is why the effective fixes are usually image size and a warm minimum replica count rather than autoscaler tuning. If the workload cannot tolerate the total, the answer is to keep a node warm — an over-provisioning Deployment of low-priority pause pods that a real workload preempts is the standard pattern, and PriorityClasses for inference covers the priority mechanics it depends on.

Diagnosing a group that will not grow

1. Read the status ConfigMap: kubectl -n kube-system describe configmap cluster-autoscaler-status. It lists every node group with its current, minimum and maximum size and its health, which immediately separates “at maximum” from “not considered”.
2. Read the pod’s events. The autoscaler writes back to the Pending pod: pod didn’t trigger scale-up followed by the reason per group is the most informative single line in this whole area.
3. Check the group is at neither its maxSize nor a hard cloud quota. An account limit on accelerator instances presents exactly like a misconfigured autoscaler; the cloud provider’s quota console is the place to confirm the limit and its exact name, and those are values worth reading rather than assuming.
4. Verify the node-template tags on the group if it is at zero. This is the cause in most “works when there is one node, never starts from empty” reports.
5. Confirm the pod could actually run on the node the template describes. A pod requesting more GPUs than one node has, or selecting a label the template lacks, is unschedulable by construction and no amount of scaling helps. The rest of that diagnosis is on the insufficient-GPU fix page.

If most of this configuration feels like telling the autoscaler things it could have discovered, Karpenter takes the other approach and derives node shapes from the cloud provider’s instance catalogue instead of from tags you maintain.