Kubernetes for Model Serving

Most of Kubernetes transfers unchanged to model serving. Three things do not: GPUs are allocated as whole integers and cannot be overcommitted, a pod can take minutes to become useful after it starts, and killing a pod mid-generation destroys work the user is watching. Every unusual field in the manifest below exists for one of those three.

A GPU is not like a CPU share

CPU and memory are native resources the scheduler understands intimately: CPU is compressible and shareable in fractions, memory is incompressible but still divisible. A GPU arrives through a device plugin as an extended resource, conventionally named nvidia.com/gpu, and it obeys different rules.

• It must be an integer. There is no nvidia.com/gpu: 500m. You get whole devices.
• Request and limit must be equal. Extended resources cannot be overcommitted, so specifying only a limit sets the request to match, and specifying different values is rejected. A pod asking for a GPU is therefore always in the Guaranteed QoS class with respect to that device.
• Two pods do not share one device by default. Sharing requires an explicit mechanism — the hardware partitioning some data-centre cards support, or a time-slicing configuration in the device plugin. Both exist; neither is on unless you turned it on, and each has real caveats about memory isolation.
• Unschedulable is the normal failure. If no node has a free device, the pod sits in Pending indefinitely. It does not degrade to CPU. It waits.

The practical consequence is that your replica count is bounded by your device count in a way that CPU services never are, and that bound is the first thing to check when a rollout stalls.

The Deployment, annotated

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-server
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0      # never drop below 3 healthy pods
      maxSurge: 1            # needs one spare GPU to roll at all
  selector:
    matchLabels: { app: llm-server }
  template:
    metadata:
      labels: { app: llm-server }
    spec:
      terminationGracePeriodSeconds: 180
      nodeSelector:
        accelerator: nvidia-a100
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      volumes:
        - name: model-cache
          hostPath: { path: /var/lib/model-cache, type: DirectoryOrCreate }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 8Gi }
      containers:
        - name: server
          image: registry.example.com/llm-server:2026.08.04
          ports:
            - { name: http, containerPort: 8000 }
          env:
            - { name: MODEL_ID,  value: "my-model@sha256:abc123" }
            - { name: CACHE_DIR, value: "/cache" }
          resources:
            requests:
              cpu: "8"
              memory: 32Gi
              nvidia.com/gpu: 1
            limits:
              cpu: "16"
              memory: 32Gi
              nvidia.com/gpu: 1
          volumeMounts:
            - { name: model-cache, mountPath: /cache }
            - { name: shm,         mountPath: /dev/shm }
          startupProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 10
            failureThreshold: 60        # up to 10 minutes to load weights
          readinessProbe:
            httpGet: { path: /ready, port: http }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 20
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]

maxSurge: 1 with maxUnavailable: 0 is the safe rolling policy, and on GPU nodes it has a hard prerequisite: a spare device must exist somewhere in the cluster, or the surge pod goes Pending and the rollout blocks forever with the old pods still healthy. If you run your GPU pool at exactly one hundred per cent allocation, invert the policy — maxUnavailable: 1, maxSurge: 0 — and accept the capacity dip during a deploy.

Memory request equals memory limit deliberately. Host RAM holds the weights while they are being copied to the device and holds the tokeniser, the request buffers and the CUDA host allocations throughout; being evicted under node memory pressure halfway through loading a 40 GB model wastes several minutes. CPU has a limit above its request because tokenisation and HTTP handling are bursty and throttling them makes the GPU wait.

The /dev/shm volume is not decoration. The default shared-memory size in a container is small, and multi-process data loaders and some multi-GPU communication paths use it. The symptom when it is too small is a bus error or a cryptic dataloader crash under load, never at startup.

Three probes, three different questions

The most common Kubernetes mistake in model serving is using one probe for all three questions. They are genuinely different.

Make the readiness endpoint mean something. A handler that returns 200 unconditionally is worse than no probe, because it converts an observable failure into a silent one. A useful /ready checks that the weights are resident, that the queue depth is below its admission bound, and that the last inference completed without a device error.

Draining a pod that is mid-stream

When a pod is deleted, two things happen concurrently: it is removed from Service endpoints, and its container receives SIGTERM. Concurrently is the problem — endpoint removal propagates through kube-proxy and any ingress controller asynchronously, so requests can still arrive for a second or two after the signal. That is what the preStop sleep is for. It delays the signal long enough for endpoint removal to land, and it is the standard fix for the handful of connection-refused errors that appear during every deploy.

Then the grace period has to be long enough for real work. terminationGracePeriodSeconds: 180 is a statement that no single generation should exceed three minutes. Your server must cooperate: on SIGTERM, stop accepting new requests, let in-flight generations finish, then exit. If it exits immediately on the signal you have a fast, clean deploy that truncates every active stream.

# The shape of a cooperative shutdown, framework-independent.
import asyncio, signal

shutting_down = asyncio.Event()

def on_term(*_):
    shutting_down.set()          # readiness starts failing; queue stops accepting

signal.signal(signal.SIGTERM, on_term)

# In the admission path:
#   if shutting_down.is_set(): return 503 with Connection: close
# In the shutdown path:
#   await inflight.wait_until_empty(timeout=grace_seconds - 15)

Budget the grace period as: preStop sleep, plus the longest generation you allow, plus a margin. If the sum exceeds what your platform will wait during a node drain or a spot reclamation, the platform wins and streams are cut anyway — spot and preemptible GPUs works through that case.

Getting pods onto the right nodes

GPU nodes are usually tainted so that ordinary workloads do not land on them and waste an expensive machine on a log shipper. Your pod therefore needs a matching toleration — and a toleration only permits scheduling, it does not attract. To require a particular class of device you also need a nodeSelector or node affinity on a label the node actually carries. The device plugin and node feature discovery typically publish labels describing the product name, memory and driver version; check what your cluster publishes with kubectl get nodes --show-labels rather than guessing the key.

Two more objects are worth adding on day one. A PodDisruptionBudget stops a node drain from taking all your replicas at once. And a pod anti-affinity across nodes keeps replicas from stacking onto one machine, which matters more than usual here because losing one GPU node can otherwise mean losing your whole serving capacity.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-server
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: llm-server }

The four failures you will actually hit

1. Pod stuck in Pending. kubectl describe pod and read the events. Insufficient nvidia.com/gpu means no free device — check whether the device plugin DaemonSet is running on every GPU node, because a crashed plugin makes the devices invisible while the hardware is fine.
2. CrashLoopBackOff with no application error. Almost always a liveness probe firing during model load. Add or lengthen the startup probe; the number to use is your observed cold load time plus fifty per cent.
3. OOMKilled during load. Container memory limit, not GPU memory. Loading weights transiently needs host RAM roughly the size of the weights on top of the steady-state footprint. Raise the limit or stream the load rather than reading the file whole.
4. CUDA out of memory under load, but not at start. This is device memory and the limit is the card, not Kubernetes. Cap concurrent sequences and KV-cache size in the server, since the cache grows with batch size and context length — the KV cache explains the growth, and VRAM requirements gives the sizing arithmetic.