GPU Time-Slicing on Kubernetes

Time-slicing lets ten pods schedule onto one GPU. It does not give those pods a tenth of the GPU each. Understanding the difference is the whole of this feature, because the config is four lines and the consequences are not.

What a replica actually is

The device plugin advertises an integer count of nvidia.com/gpu. Time-slicing changes that integer. With replicas: 4 on a node with two physical GPUs, the node advertises eight, and the scheduler will place eight single-GPU pods there. Each pod is handed a real device; four pods share each physical card, and the GPU’s own scheduler interleaves their kernels in time.

NVIDIA states the limitation directly in its GPU Operator documentation: unlike Multi-Instance GPU, there is no memory isolation and no fault isolation between replicas, but for some workloads it is better than not being able to share at all. Both halves of that sentence matter. Concretely:

• Memory is first-come. Four replicas share one memory pool. A pod that loads a model filling most of the card leaves the other three to fail on allocation. Nothing in the resource model prevented that scheduling decision, because the scheduler was counting replicas, not bytes.
• A fault takes everything down. An Xid error or a wedged context on the physical GPU affects every pod sharing it, not just the one that caused it.
• Throughput is divided, not multiplied. Time-slicing improves utilisation of a card that is idle between requests. It cannot create arithmetic. Four saturating workloads on one GPU each run at roughly a quarter speed plus context-switch overhead.

The workloads it suits are the bursty, small ones: notebooks, development environments, low-traffic model endpoints, batch scoring jobs that spend most of their wall time on I/O. The workload it does not suit is production inference with a latency target.

The plugin configuration

The configuration is a YAML document handed to the device plugin, withsharing.timeSlicing as the relevant stanza:

apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
  namespace: nvidia-device-plugin
data:
  shared-4: |-
    version: v1
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4

The two booleans decide how much of this is visible to the people writing pod specs.

• renameByDefault: true advertises the replicas as nvidia.com/gpu.shared instead of nvidia.com/gpu. This is the safer setting on a shared cluster: a workload that must have a whole card keeps asking for nvidia.com/gpu and cannot accidentally receive a slice. The cost is that every existing shared manifest has to change its resource name.
• failRequestsGreaterThanOne: true makes a request for more than one replica fail rather than silently succeed. Leave it on. Two replicas are not two GPUs and may well be two slices of the same physical card, so a pod that asks for two and expects to shard a model across them gets something that looks right and performs wrongly.

Applying it to a subset of nodes

A cluster rarely wants uniform slicing. The plugin supports several named configurations in one ConfigMap and picks per node from the label nvidia.com/device-plugin.config, which is how one pool of A100s can be whole-card while a pool of smaller cards is sliced four ways.

1. Create the ConfigMap above with one key per profile — here, shared-4. Add a default key if you want a cluster-wide fallback.
2. Point the plugin at it. With the Helm chart that is --set config.name=device-plugin-config on the nvdp/nvidia-device-plugin release.
3. Label the nodes that should use the profile: kubectl label node gpu-node-1 nvidia.com/device-plugin.config=shared-4. Nodes without the label keep the default.
4. Wait for the plugin on that node to restart and re-advertise. The allocatable count is the confirmation, not the pod status.

Verifying and then breaking it

Confirm the count first:

kubectl get node gpu-node-1 -o jsonpath='{.status.allocatable}' | tr ',' '\n' | grep nvidia

A node with two physical GPUs and replicas: 4 should report 8. Then schedule more single-GPU pods than there are physical cards and watch them all reach Running — the thing that was impossible before.

Now do the test that matters, which is the failure. Give one of those pods a workload that allocates most of the card’s memory and watch what happens to its neighbours. They will not be evicted, they will not be rescheduled, and Kubernetes will report nothing wrong: they will fail inside the CUDA runtime with an out-of-memory error while the pod remains Running and Ready. That asymmetry — a resource exhausted with no Kubernetes-level signal — is why time-slicing needs to be a deliberate decision per node pool rather than a default.

Choosing a replica count

There is no documented maximum, which is not permission to set it high. The useful upper bound is memory: divide the card’s memory by the largest resident footprint you will let onto it, and do not exceed that number, because every replica beyond it is a scheduling slot that cannot actually run. For inference servers the footprint is the model weights plus the KV cache allocation, and the KV cache is typically configured as a fraction of total GPU memory — which means two replicas of a server left at its default fraction will collide by design. Cap that fraction explicitly on every sliced pod.

If the workloads genuinely need isolation rather than density, time-slicing is the wrong mechanism and MIG partitioning is the right one: it divides memory and compute in hardware, at the cost of fixed partition sizes and supported-GPU requirements. And if the goal is simply to stop half-empty GPUs accumulating, bin-packing GPU pods addresses that without sharing a card at all.