Draining a GPU Node Without Killing In-Flight Requests

A generation request can take ninety seconds. The default termination grace period is thirty. Everything unpleasant about draining a GPU node follows from those two numbers, plus one asynchronous step that is not obvious from any manifest.

What a drain actually does

kubectl drain is two operations. First it cordons the node — marks it unschedulable so nothing new lands there. Then it evicts every pod that is not a DaemonSet or a mirror pod, using the Eviction API, which respects PodDisruptionBudgets. Eviction is a deletion: the API server marks the pod for deletion, the kubelet sends SIGTERM to the container’s process 1, and after terminationGracePeriodSeconds it sends SIGKILL.

On a GPU node there is a wrinkle worth knowing before you start. The NVIDIA device plugin runs as a DaemonSet, and kubectl drain refuses to proceed with DaemonSet-managed pods present unless you pass --ignore-daemonsets. That flag does not evict them — it skips them — which is what you want, because the plugin should be the last thing running on a GPU node, not the first thing removed.

What kubectl drain waits for is narrower than most people assume, and the gap is where in-flight requests die. It waits for pods to be deleted from the API server. It does not wait for an in-flight HTTP response to finish, because it has no way to know one exists; it does not wait for a proxy somewhere else in the cluster to stop routing to the pod; and it does not wait for a replacement pod to become Ready on another node. A drain can therefore return successfully while your service has fewer healthy replicas than it did a minute ago and a handful of users have just seen a truncated stream.

That is why terminationGracePeriodSeconds is the parameter this whole page turns on. It is the only lever that buys the pod time between “you are going away” and “you are gone”, and drain honours it — the command blocks until each pod finishes terminating or its grace period expires. Set it shorter than your longest generation and drain will hand you a clean exit code while SIGKILL cuts responses mid-token; set it longer, and the same command waits.

The race nobody sees until it bites

When a pod is deleted, two things happen concurrently and nothing orders them. The kubelet begins termination and sends SIGTERM. Separately, the endpoints controller observes the deletion and removes the pod’s address from the Service’s EndpointSlice, after which every kube-proxy or ingress controller in the cluster must observe that and update its own forwarding rules.

The second path takes time — several control-plane hops, each eventually consistent — and it is entirely possible for the server to receive SIGTERM, stop accepting connections, and still be receiving new requests from a proxy that has not caught up. Those requests get a connection refused, which surfaces to your users as a 502 during what you thought was a graceful rollout.

Lengthening the grace period does not fix this. The grace period governs how long the process has after SIGTERM, and the problem is that SIGTERM arrived before the world stopped sending traffic. What fixes it is delaying the signal.

preStop and the grace period

Kubernetes documents that the preStop hook runs before SIGTERM is sent and blocks it: the hook must complete before the TERM signal can be sent. It also documents that the grace-period countdown starts before the hook runs and covers both the hook and the shutdown, so the two durations share one budget. A hook of 55 seconds plus a shutdown of 10 under a 60-second grace period means the container is killed before it finishes.

spec:
  terminationGracePeriodSeconds: 150
  containers:
    - name: server
      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - -c
              - |
                # Fail readiness immediately so the endpoints controller
                # removes this pod, then wait for proxies to catch up.
                touch /tmp/shutting-down
                sleep 20
      readinessProbe:
        exec:
          command: ["/bin/sh", "-c", "! test -f /tmp/shutting-down && curl -sf localhost:8000/health"]
        periodSeconds: 2
        failureThreshold: 1

Read the budget as a sum: 20 seconds of preStop plus however long the server needs to finish its longest in-flight generation, all inside 150. If your p99 generation is 90 seconds, 150 is about right and 60 is not.

A sleep lifecycle handler exists as a first-class alternative to the exec form, so a container without a shell can still delay. It went through beta before graduating, so it is available on any reasonably current cluster but not on an old one — check your version before depending on it.

      lifecycle:
        preStop:
          sleep:
            seconds: 20

The server itself must also do its part: on SIGTERM it should stop accepting new connections and let existing ones finish, rather than exiting immediately. A server that calls exit(0) on SIGTERM makes every hook and grace period above irrelevant.

A disruption budget for the fleet

Everything so far protects requests on one pod. A PodDisruptionBudget protects the service while several nodes are drained in sequence, which is what a cluster upgrade is.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inference
  namespace: inference
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: mistral-7b

The Eviction API refuses an eviction that would breach the budget, so kubectl drain blocks and retries rather than proceeding. That is the intended behaviour and it is also the classic way to wedge a cluster upgrade: a budget of minAvailable: 2 on a Deployment with two replicas means no pod may ever be evicted, and the drain hangs forever. Express the budget as maxUnavailable: 1 instead when the replica count is small, so it scales with the deployment rather than fighting it.

A budget also only helps if a replacement can actually be placed. On a GPU cluster running near capacity, evicting one replica does not reliably produce another — the recreated pod needs a free device, and if there is none it sits Pending while the budget counts the terminating pod as gone. The result is a drain that satisfies the budget on paper and halves your serving capacity in practice. Draining a GPU node safely usually means growing the pool by one node first, so there is somewhere for the evicted replicas to land, and shrinking it afterwards.

Running the drain

1. Cordon first, separately, and watch: kubectl cordon gpu-node-3. Nothing is evicted yet. If the service is autoscaled, this is the moment to let it add a replica elsewhere before you take capacity away.
2. Drain with the flags a GPU node needs: kubectl drain gpu-node-3 --ignore-daemonsets --delete-emptydir-data --timeout=15m. The emptydir flag is required because model caches and /dev/shm are emptyDir volumes, and drain refuses to delete pods with local data unless told to.
3. Do not reach for --force or --disable-eviction when it blocks. --force deletes pods with no controller; --disable-eviction bypasses PodDisruptionBudgets entirely and deletes directly, which discards the protection you configured. If drain is blocked, read kubectl get pdb -A and fix the budget.
4. Confirm the node is empty of workload pods before you act on it: kubectl get pods --field-selector spec.nodeName=gpu-node-3 -A. Only DaemonSet pods should remain.
5. Afterwards, kubectl uncordon gpu-node-3. A forgotten cordon on a GPU node is expensive and invisible — the node reports Ready and takes no work.