Scaling GPU Nodes to Zero With Karpenter

Karpenter provisions a node shaped for the pods that are Pending rather than adding one from a pre-declared group. For GPUs that removes the usual scale-from-zero problem — it does not need to be told what an empty node group would have contained.

How Karpenter differs from a node group

The classic cluster autoscaler grows and shrinks node groups you defined in advance, and when a group is at zero it has to be told, via tags, what a node in it would have looked like. Karpenter has no groups. It watches unschedulable pods, computes an instance type that would fit them from the constraints you gave it, and launches one.

For GPU workloads this matters twice. Scaling from zero needs no resource hints, because Karpenter already knows what every instance type offers. And a pod requesting a GPU can be satisfied from a range of instance families, so a capacity shortage in one does not stall the workload — provided you expressed the constraint as a range rather than as one type.

The GPU NodePool

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        workload: inference
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu
      taints:
        - key: nvidia.com/gpu
          value: "present"
          effect: NoSchedule
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["g5", "g6"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
  limits:
    nvidia.com/gpu: 16

The taints block is doing real work here. Karpenter is taint-aware: it only launches a node from this pool for pods that tolerate the taint, so the taint both keeps CPU workloads off the expensive nodes and stops Karpenter provisioning GPU nodes for pods that never needed one. Without it, a burst of Pending CPU pods can pull up a GPU instance because it satisfies their CPU request. The toleration side is on taints and tolerations for a GPU node pool.

limits is the cap that keeps a runaway autoscaler from provisioning an unbounded number of accelerators, and expressing it in nvidia.com/gpu rather than in CPU is the version that means something for this pool. It is the single most valuable line in the manifest for anybody with a budget.

expireAfter is documented with a default of 720h. On GPU nodes that is a node lifetime of a month, which is long enough to matter for driver and AMI patching; shortening it forces a rolling replacement.

The EC2NodeClass

The NodePool says what kind of node; the EC2NodeClass says how to build it. Karpenter requires amiSelectorTerms, subnetSelectorTerms, securityGroupSelectorTerms and one of role or instanceProfile.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu
spec:
  role: KarpenterNodeRole-my-cluster
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 200Gi
        volumeType: gp3
        deleteOnTermination: true

Two GPU-specific notes. Karpenter’s AMI aliases resolve to an image appropriate for the instance type, so an accelerated instance launched from a family alias gets the accelerated variant rather than needing a separate alias — but pinning @latest means the AMI can change under you, and a specific version is the safer choice for a GPU pool where driver versions are load-bearing. And the volume size is not boilerplate: inference images with CUDA runtimes and model weights baked in routinely exceed a default root volume, and the failure mode is an image pull that fills the disk and evicts pods. If weights are pulled at runtime rather than baked in, size for those too — baking versus mounting model weights is the decision behind that number.

Karpenter does not install the NVIDIA device plugin. A node it provisions has GPUs and advertises none until the DaemonSet lands on it, which is a scheduling race on every new node — see installing the device plugin.

Getting back to zero

Provisioning is the easy half. Karpenter’s documented consolidation policies are WhenEmpty, WhenEmptyOrUnderutilized and Balanced, and the choice decides whether an idle GPU node ever goes away.

• WhenEmpty removes a node only once nothing is running on it. Conservative, and on a GPU pool it means one leftover pod keeps an expensive instance alive indefinitely.
• WhenEmptyOrUnderutilized also removes or replaces nodes whose pods would fit elsewhere, which is what actually returns a pool to zero as demand falls.
• consolidateAfter is the idle period before acting, documented with a default of 1m and accepting Never. On GPU nodes a longer value is usually right: if reacquiring the node costs several minutes of cold start, thrashing it after sixty seconds of quiet is a bad trade.

Two things will silently prevent consolidation. A pod with no controller cannot be rescheduled, so Karpenter will not disrupt its node; and a PodDisruptionBudget that cannot be satisfied blocks eviction. On a GPU pool, either one is an instance that never goes away, so audit both before concluding the policy is broken. The annotation karpenter.sh/do-not-disrupt on a pod does the same thing deliberately, which is the right tool for a long-running job that must not be interrupted mid-flight.

When the instance is not available

GPU instance types are the ones most likely to be unavailable in a given zone at a given moment. Karpenter records launch failures and will try alternatives it is permitted to use, so the practical defence is to widen the requirement: several instance families, several zones, and where the workload tolerates interruption, karpenter.sh/capacity-type including spot — covered on spot GPU node pools. A NodePool pinned to one family in one zone has no alternative to try, and the symptom is pods Pending with no scale-up and nothing obviously wrong.