Spot Node Pools for Kubernetes GPU Workloads

A spot GPU node is the same hardware with a different contract: you get it cheaper and the provider can take it back with a short warning. The Kubernetes work is making sure only workloads that can survive that land on it, which is a taint, and making sure they actually get there, which is a toleration plus affinity.

What you are actually agreeing to

Every provider’s spot offering has the same two properties: a reduced price, and reclamation with a notice period measured in seconds to a couple of minutes. Read the second one as an operational fact rather than an edge case. A GPU pool of ten spot nodes will lose nodes, and a batch job that checkpoints every four hours on a twenty-minute-lifetime node makes no progress at all while still costing money.

The discount and the notice period are provider- and region-specific and they move; take both from the provider’s own current pricing and documentation rather than from any figure quoted in a blog post. What does not move is the shape of the design: the workload must tolerate being killed mid-request, and something must decide what happens to the traffic that was in flight.

The taint and label keys, per platform

This is the part worth getting from a primary source, because the keys are not guessable and two of the three platforms apply the taint for you.

• GKE. Google documents that GKE automatically adds the taint cloud.google.com/gke-spot=true:NoSchedule to nodes in new Spot VM node pools, and labels those nodes cloud.google.com/gke-spot=true — with cloud.google.com/gke-provisioning=spot as an additional label on recent versions. See Google’s Spot VMs in GKE documentation.
• AKS. Microsoft documents the spot pool as carrying the label kubernetes.azure.com/scalesetpriority:spot and the taint kubernetes.azure.com/scalesetpriority=spot:NoSchedule, applied when the pool is created. Microsoft also documents that a spot pool cannot be the cluster’s system node pool. See Microsoft’s AKS spot node pool documentation.
• EKS with Karpenter. Capacity type is a requirement key on the NodePool, karpenter.sh/capacity-type, with values spot and on-demand. Karpenter does not taint spot nodes for you — if you want the isolation, you write the taint into the NodePool’s spec.template.spec.taints. See the Karpenter NodePool documentation.

Creating the pool

# GKE — the spot taint is added for you.
gcloud container node-pools create gpu-spot \
  --cluster inference --region europe-west4 \
  --spot \
  --accelerator type=nvidia-l4,count=1 \
  --machine-type g2-standard-8 \
  --enable-autoscaling --min-nodes 0 --max-nodes 8

# AKS — priority Spot, with the eviction policy stated explicitly.
az aks nodepool add \
  --resource-group inference-rg --cluster-name inference \
  --name gpuspot --priority Spot --eviction-policy Delete \
  --spot-max-price -1 \
  --node-vm-size Standard_NC24ads_A100_v4 \
  --enable-cluster-autoscaler --min-count 0 --max-count 8

# EKS with Karpenter — you write the taint yourself.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-spot
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g6.xlarge", "g6.2xlarge", "g5.xlarge"]
      taints:
        - key: workload-class
          value: interruptible
          effect: NoSchedule
  limits:
    nvidia.com/gpu: 16
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized

The instance-type list in the Karpenter block is not padding. Spot availability is per instance type per zone, so a NodePool that will accept three GPU types is materially more likely to get capacity than one that insists on a single type — and it is the cheapest resilience available here.

The workload side: toleration plus affinity

A taint keeps pods off. It does not pull pods on. Google’s guidance is explicit that a deployment targeting spot needs both the matching toleration and a node affinity rule on the spot label — without the affinity, the pod is merely allowed on spot nodes and will happily take an on-demand one instead, which is the exact opposite of the saving you were trying to make.

      tolerations:
        - key: cloud.google.com/gke-spot
          operator: Equal
          value: "true"
          effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: cloud.google.com/gke-spot
                    operator: In
                    values: ["true"]

Make the affinity preferred rather than required if you want the workload to fall back to on-demand when spot capacity is exhausted. That is a real decision with a price attached, and it should be made per workload rather than by default.

Surviving the reclaim

1. Set a grace period you can actually use. The reclaim notice is short and the provider will not wait for a slow shutdown, so a 120-second grace period on a node with a thirty-second warning buys nothing. Match the pod’s termination behaviour to the notice, not to your preference.
2. Stop taking new requests immediately. A preStop hook that fails the readiness probe and then sleeps gives the endpoints controller time to remove the pod before the server stops accepting connections — the mechanism in draining a GPU node gracefully.
3. Run a node-termination handler. The reclaim signal arrives on a provider metadata endpoint, not through the Kubernetes API. Something must watch it and cordon and drain the node; on AWS that is the node termination handler or Karpenter’s own interruption handling, and the equivalent exists on the other platforms.
4. Spread replicas. Spot reclamation frequently hits a whole instance type in a zone at once. A topologySpreadConstraint across zones is what stops one reclaim event taking every replica of a service simultaneously.

What must not go here

Anything holding state that is expensive to rebuild: a vector index being constructed, a fine-tuning run without checkpointing, a queue consumer that acknowledges before it finishes. Anything with a latency SLO measured in a single-digit number of seconds, because a reclaim during model load means minutes of unavailability for that replica. And the last replica of anything — a spot pool is a place to put capacity you can afford to lose, and if losing it is an outage then the pool is not doing the job you bought it for.

There is also a class of workload that is technically interruptible and still a poor fit: anything whose startup cost is large relative to its expected lifetime. A model server that takes six minutes to pull weights and load them onto the device, running on nodes that are reclaimed every forty minutes, spends a substantial fraction of its existence not serving anything while costing you the node the whole time. The discount has to beat that overhead, and on a large model it frequently does not. The fix, where you want to use spot for serving anyway, is to attack the startup cost first — weights on a node-local cache or a shared volume rather than a fresh download per pod — and only then measure whether the arithmetic works.

Finally, do not run cluster-critical infrastructure on the same pool. It is tempting to let a spot pool host the ingress controller or the metrics adapter because they are small and the nodes are cheap. A reclaim event that takes out the component your autoscaler depends on arrives at exactly the moment you need scaling to work, and this is why managed platforms exclude spot pools from system workloads by default rather than as a suggestion.