Resource Requests and Limits for a GPU Inference Pod

You can ask for 500 millicores of CPU and 1.5GiB of memory. You cannot ask for half a GPU. The reason is one rule about extended resources, and almost every other GPU scheduling behaviour is a consequence of it.

The rule everything follows from

nvidia.com/gpu is an extended resource: a resource type outside the built-in cpu, memory, ephemeral-storage and hugepages-* set, advertised by a device plugin. Kubernetes applies three rules to extended resources, and they are not configurable:

• Integer quantities only. There is no fractional form. 0.5 is rejected.
• Request must equal limit. The GPU documentation states you may give a limit alone, in which case the request is set from it; you may give both if they are equal; you may not give a request alone.
• No overcommitment. The sum of requests on a node cannot exceed allocatable. This is also why a ResourceQuota for an extended resource only accepts the requests. prefix — with request and limit identical and no overcommitment, a separate limits key would be a second name for the same number.

Why CPU can be divided and a GPU cannot

The difference is enforcement, not arithmetic. A CPU request is a scheduling weight and a CPU limit is enforced by the kernel through cgroup CPU bandwidth control: the kernel really can give a container exactly 500 millicores by throttling it. Memory is enforced by the cgroup memory controller, which kills the process at the limit. Kubernetes can divide these because Linux can divide these.

A GPU has no equivalent in the container runtime. What the device plugin does at allocation time is tell the runtime to make a particular device visible to a particular container — a whole device, by UUID. There is no kernel mechanism that will cap a container at half a card’s throughput or half its memory. So Kubernetes offers the only honest accounting available: countable whole devices, allocated exclusively.

This explains the behaviour that most often reads as a bug. A GPU pod that is completely idle still holds its device, because the reservation is an allocation and not a measurement. Nothing in Kubernetes will reclaim it, no scheduler will oversubscribe it, and kubectl top will never show GPU utilisation, because it reports from the metrics pipeline for CPU and memory and knows nothing about accelerators. GPU utilisation comes from an exporter reading NVML, on a separate path — see GPU monitoring.

What it does to QoS class

Quality of Service class is computed from requests and limits across all containers in a pod, and it decides eviction order under node pressure. Because a GPU request always equals its limit, the GPU resource pushes the pod towards Guaranteed — but only if every other resource in every container also has equal request and limit. Set a memory request of 8Gi with a limit of 16Gi alongside a GPU, and the pod is Burstable.

For an expensive, slow-starting GPU pod, Guaranteed is usually what you want: BestEffort pods are evicted first under node memory pressure, then Burstable pods that exceed their requests, and Guaranteed last. Evicting a model server to save a logging agent is a bad trade, and it is a trade you make implicitly by leaving the memory limit off. A PriorityClass is the complementary control for preemption rather than eviction, covered on PriorityClasses for inference.

The CPU and memory you still have to set

The GPU line is the one people write and the other two are the ones that cause incidents. An inference container needs host memory for more than bookkeeping: weights are frequently read into host memory before being copied to the device, the CUDA runtime pins buffers there, and tokenisation and request handling all run on the CPU.

    resources:
      requests:
        cpu: "4"
        memory: 24Gi
      limits:
        cpu: "8"
        memory: 24Gi
        nvidia.com/gpu: 1

Memory request equals limit deliberately: exceeding a memory limit is an immediate OOMKill, and on a container that takes minutes to load weights, an OOMKill during a traffic spike is a very long outage. Sizing it from steady-state usage alone tends to be too tight, because peak host memory usually occurs during loading rather than during serving.

The CPU limit is left above the request, which is the one place bursting is worth having. Tokenisation and HTTP handling are bursty and CPU throttling shows up as latency on a request path where the GPU is otherwise ready to work. Omitting the CPU limit entirely is defensible on a dedicated GPU node pool, where there is nothing else to protect — but it forfeits Guaranteed class, so it is a choice rather than a default.

The two ways to buy a fraction anyway

Both work by changing what the node advertises, never by changing what the pod requests. The pod always asks for an integer number of whatever the node is offering.

• Time-slicing advertises each physical GPU as several replicas. The count goes up, the isolation does not exist: NVIDIA documents that there is no memory or fault isolation between replicas. Two pods can each be handed a “GPU” and then fight over the same memory. GPU time-slicing has the configuration.
• MIG advertises hardware-isolated partitions, either under nvidia.com/gpu or under profile-specific names. Real memory isolation, at the cost of fixed profile sizes and Ampere-or-later hardware. MIG partitioning has the strategies.

If neither mechanism fits, the remaining lever is the model rather than the scheduler: a smaller or more heavily quantised model that leaves room for a second workload on the same card is often a better answer than partitioning, because it removes the contention instead of managing it.