Namespace Resource Quotas for a Shared GPU Cluster

A shared GPU cluster fails in one predictable way: one team’s experiment consumes every device and everyone else’s pods go Pending. A ResourceQuota is the control that prevents it, and there is one syntax rule for GPUs that is not optional.

The rule for extended resources

nvidia.com/gpu is an extended resource, not a core one, and extended resources behave differently in quota. The Kubernetes documentation states it directly: because overcommit is not allowed for extended resources, specifying both requests and limits for the same extended resource in a quota makes no sense, so only quota items with the requests. prefix are allowed. See the Kubernetes resource quota documentation.

So requests.nvidia.com/gpu: "8" is valid and limits.nvidia.com/gpu is not. This is the single most common mistake in GPU quota manifests, and it is not silently ignored — the API server rejects the ResourceQuota object, which at least fails loudly. The counterpart rule on the pod side is that an extended resource’s request must equal its limit, which is why quoting one of them is sufficient to bound the other.

It is worth being precise about what the number means, because the word “quota” suggests a reservation and it is not one. A quota of eight GPUs does not set aside eight devices for that namespace. It caps the sum of GPU requests across the namespace’s non-terminal pods, and nothing more. A team can hold quota it cannot use, if the cluster has no free devices, and the sum of every namespace’s quota can exceed the number of GPUs the cluster physically has — usually it should, or you will strand capacity. Quota bounds demand; the scheduler still decides supply.

The quota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: team-search
spec:
  hard:
    requests.nvidia.com/gpu: "8"
    requests.cpu: "96"
    requests.memory: 512Gi
    limits.cpu: "192"
    limits.memory: 768Gi
    count/deployments.apps: "40"
    persistentvolumeclaims: "20"

Note the asymmetry: CPU and memory carry both a requests. and a limits. entry because they are compressible or overcommittable and the two numbers mean genuinely different things, while the GPU line exists once. The object-count entries are there because a namespace that cannot exhaust GPUs can still exhaust the control plane, and a runaway controller creating Deployments is a real failure mode on shared clusters.

kubectl apply -f gpu-quota.yaml
kubectl describe resourcequota gpu-quota -n team-search

Name:                     gpu-quota
Namespace:                team-search
Resource                  Used  Hard
--------                  ----  ----
requests.cpu              24    96
requests.memory           128Gi 512Gi
requests.nvidia.com/gpu   3     8

The Used column is the sum over non-terminal pods in the namespace. It is the number to put on a dashboard: a namespace sitting at its GPU hard limit for days is either under-provisioned or hoarding, and you cannot tell which from the scheduler’s events alone.

What happens when it is exceeded

Quota is enforced at admission, not at scheduling. A pod that would push the namespace over its GPU quota is rejected by the API server when it is created — it never becomes a Pending pod, and it never appears in kubectl get pods.

That distinction is the source of the most confusing symptom in this area. If a Deployment is scaled beyond quota, the Deployment reports the desired replica count and the pods simply do not exist. The error is on the ReplicaSet, not on any pod:

kubectl describe replicaset team-search-ranker-7c9b4

Events:
  Type     Reason        Age   From                   Message
  ----     ------        ----  ----                   -------
  Warning  FailedCreate  12s   replicaset-controller  Error creating: pods
    "team-search-ranker-7c9b4-" is forbidden: exceeded quota: gpu-quota,
    requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=8,
    limited: requests.nvidia.com/gpu=8

The message names the quota object, the request, the current usage and the cap. When someone reports that “the deployment is not creating pods”, this is the first place to look, and kubectl describe replicaset — not describe deployment — is the command that shows it.

The controller keeps retrying with backoff, so the moment quota frees up the missing replicas appear without anyone intervening. That is usually what you want and it is occasionally a surprise: a Job that was rejected an hour ago can start the instant another team’s workload finishes, in the middle of the night, on hardware somebody else had earmarked. If a workload must not start opportunistically, quota is the wrong control for it — suspend the Job or gate it in the pipeline.

Two consequences of admission-time enforcement are worth holding on to. First, an existing pod is never evicted when a quota is lowered; the namespace simply sits over its cap until pods churn naturally, and Used can legitimately exceed Hard for a while. Second, a pod that has already been admitted counts against quota for its whole lifetime including while it is terminating, so a rolling update needs quota headroom for the surge — a namespace pinned at exactly its GPU cap cannot roll a Deployment at all, because the new pod cannot be created until the old one is gone and the old one will not go until the new one is Ready. That deadlock reads as a stuck rollout with no scheduling error anywhere, and the fix is either one GPU of headroom in the quota or maxSurge: 0 on the Deployment.

Quota needs a LimitRange to work

There is a trap in how quota interacts with pods that specify nothing. If a namespace has a quota on requests.cpu, every pod in it must declare a CPU request or its creation is rejected. Teams hit this immediately, and the fix is a LimitRange supplying defaults so that existing manifests keep working.

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: team-search
spec:
  limits:
    - type: Container
      default:
        cpu: "2"
        memory: 4Gi
      defaultRequest:
        cpu: "1"
        memory: 2Gi
      max:
        nvidia.com/gpu: "4"

The max entry is doing something the quota cannot: it caps a single container. A quota of eight GPUs is satisfied by one pod asking for all eight, which is a scheduling problem for everyone else even though the quota is respected. The per-container maximum keeps any one workload to a size the cluster can actually place.

A LimitRange max is also the only place to express a per-container GPU ceiling at all — there is no quota field for it — and it is enforced at admission with a clear message naming the container and the limit. Set it to the largest device count a single node in the cluster actually has. A pod asking for more GPUs than exist on any one node is unschedulable by construction, and it is much cheaper to reject it at creation than to explain a permanently Pending pod later.

Scoping a quota to a priority class

A single number per namespace is often too blunt: you want batch work capped tightly and interactive work capped loosely. Quota supports a scopeSelector keyed on priority class, so you can write two quotas in the same namespace that count different pods.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota-batch
  namespace: team-search
spec:
  hard:
    requests.nvidia.com/gpu: "4"
  scopeSelector:
    matchExpressions:
      - scopeName: PriorityClass
        operator: In
        values: ["inference-batch"]

A pod is charged against every quota whose scope it matches, so an unscoped namespace quota and a scoped batch quota compose: batch is capped at four and everything together at eight. This pairs directly with priority classes for inference, where priority decides who wins a contended GPU and quota decides how many any team may hold at once. Track the same namespace dimension in your spend reporting and the quota becomes a budget rather than just a guardrail — allocating AI infrastructure cost covers that side.

One caution on scoped quotas. If a quota exists in the namespace with a PriorityClass scope, pods must carry a priority class the cluster can resolve, and a pod naming a class that no scoped quota matches falls through to whatever unscoped quota exists — or to none. The tidy arrangement is one unscoped quota that everything is charged against, plus scoped quotas that carve it up, and a globalDefault priority class so no pod is ever unclassified. Without the default, a manifest that forgets priorityClassName silently escapes the tighter cap you thought you had applied.