A PriorityClass for Preempting Low-Priority Inference Pods

Priority does two separate things: it reorders the scheduling queue, and it authorises the scheduler to evict running pods to make room. The second is the one people want on a full GPU cluster, and it is the one with the constraint nobody mentions.

What preemption actually does

When a pod cannot be scheduled, the scheduler runs a preemption pass. For each node, it asks whether removing some set of lower-priority pods would let this pod fit. If it finds such a node, it picks victims — preferring the lowest priority, and fewest pods — and deletes them gracefully, then the pending pod is scheduled in a subsequent cycle. The victims are not moved; they are deleted, and their controllers recreate them somewhere else if there is a somewhere else.

Two consequences follow. Preemption is not instant: victims get their termination grace period, so a batch pod with a 300-second grace period delays the high-priority pod by up to 300 seconds. And preemption is not a queue — the freed space is not reserved for the pod that caused the eviction, so in principle another pod can take it, though the scheduler nominates a node on the preemptor to make this unlikely.

There is a third consequence specific to model serving, and it is the one that turns a correct configuration into an incident. Preemption takes no account of how far along a pod is. A victim that has spent four minutes pulling weights off object storage and is thirty seconds from Ready is exactly as evictable as one that started a moment ago — the scheduler sees a pod with a low priority number holding a GPU, and nothing about loading progress is expressed anywhere it can read. Evict it and that work is simply lost; when the controller recreates the pod elsewhere, the download starts from nothing.

On a busy cluster with aggressive preemption this can become a treadmill where a low-priority service never finishes starting, because it is preempted during load more often than it manages to reach Ready. The symptom is a Deployment that has been rolling for an hour with replicas cycling through ContainerCreating and Terminating and no useful error anywhere. If you see it, the answer is not more priority — it is a PodDisruptionBudget, which the preemption path respects on a best-effort basis, or moving the loading cost off the critical path with a warm cache on the node.

Defining the classes

A PriorityClass is a cluster-scoped object in scheduling.k8s.io/v1. The integer value is what matters; the name is a handle. Kubernetes documents the valid range as up to one billion for user-defined classes, with values above that reserved for the built-in system-cluster-critical and system-node-critical classes.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-interactive
value: 1000000
globalDefault: false
description: "User-facing inference. May preempt batch work."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-batch
value: 1000
globalDefault: false
description: "Offline evaluation and bulk generation. Preemptible."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: cluster-default
value: 100
globalDefault: true
description: "Everything that does not name a class explicitly."

Leave gaps between the values. You will want to insert a class between two existing ones, and renumbering means editing every workload that referenced the old name. Only one class in the cluster may set globalDefault: true; setting it defines the priority of every pod that names no class, which is the cleanest way to guarantee that “unlabelled” sits below batch rather than above it.

Applying them to workloads

apiVersion: apps/v1
kind: Deployment
metadata:
  name: chat-frontend-inference
spec:
  template:
    spec:
      priorityClassName: inference-interactive
      containers:
        - name: server
          image: registry.example.com/inference:1.9.0
          resources:
            limits:
              nvidia.com/gpu: "1"

1. Apply the classes first. A pod referencing a PriorityClass that does not exist is rejected by the API server, so ordering matters in CI.
2. Confirm the value was resolved onto the pod: kubectl get pod POD -o jsonpath='{.spec.priority}'. The admission controller writes the integer onto the pod at creation time; changing the PriorityClass afterwards does not update existing pods.
3. Watch a preemption happen. Fill the GPU nodes with inference-batch pods, then create an inference-interactive pod. The batch pod that gets evicted carries a Preempted event naming the preemptor.
4. Check the preemptor’s status.nominatedNodeName. If it is set, preemption succeeded and the pod is waiting for victims to terminate. If it is empty and the pod is still Pending, preemption found no viable node — read the next section.

Why GPUs preempt in whole units

This is the part that makes GPU preemption different from CPU preemption, and it is why a correctly configured PriorityClass can still leave your pod Pending with a message about no preemption victims being found.

nvidia.com/gpu is an integer extended resource that cannot be overcommitted, so a pod either holds a whole device or none of it. Evicting a batch pod that holds one GPU frees exactly one GPU. If your interactive pod requests two on a node whose four GPUs are held by four separate single-GPU batch pods, the scheduler must evict two of them on the same node — which it can do — but if those four pods are spread one per node across four nodes, no single node can be made to fit a two-GPU pod by any amount of eviction. The scheduler reports that preemption is not helpful, and it is right.

The same arithmetic breaks time-sliced and MIG-partitioned setups in a more subtle way, because the advertised resource units no longer map to whole devices. If you are relying on preemption to guarantee interactive latency, keep the preemptible workload at one GPU per pod, so that every eviction frees a unit the preemptor can use. Concentration also helps: the bin-packing setting in multi-GPU node bin packing makes it more likely that enough victims share a node.

Priority without preemption

Sometimes you want a workload to jump the queue but never to destroy running work — a training job that should start before other pending jobs, without killing anything already computing. Kubernetes documents preemptionPolicy: Never for exactly this: the pod is placed ahead of lower-priority pods in the scheduling queue but cannot evict anyone, and waits for resources to free naturally. It remains preemptible by higher-priority pods itself.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: training-queue-jumper
value: 500000
preemptionPolicy: Never
globalDefault: false
description: "Scheduled before other batch work, evicts nothing."

The default is PreemptLowerPriority, so omitting the field gives you eviction. On a shared GPU cluster the safest starting position is three classes — interactive with preemption, training with Never, batch as the global default — and then to add preemption only where an actual latency requirement justifies destroying somebody else’s hours of compute. Pair it with a namespace quota so that priority governs order and quota governs total consumption; the two are independent controls and neither substitutes for the other.

Two operational details close this out. Priority is not a quota — a high-priority workload with no cap can consume the whole cluster legitimately, and every other team sees only that their pods vanished. Scoping a ResourceQuota to the interactive class, as in namespace GPU quotas, is what stops privilege becoming unlimited consumption.

And make the victims survivable before you make the preemptor powerful. A batch job that checkpoints every thirty seconds loses almost nothing to an eviction; one that checkpoints hourly loses up to an hour of GPU time every time your interactive service scales up. The preemption configuration is the cheap half of this work — the expensive, valuable half is making the low-priority workload resumable, and it pays off identically when the node was a spot node that got reclaimed rather than a pod that got preempted.