Autoscaling on GPU: Metrics That Actually Work

Autoscaling an inference service on GPU utilisation does not work, and the reason is not that the metric is noisy. It is that the metric saturates: it reaches one hundred per cent while the service is still fine, and stays at one hundred per cent as the queue grows past every latency target you have. A signal that cannot distinguish healthy from failing cannot control anything. Queue depth can, and Little’s law says exactly what to set it to.

Why GPU utilisation is the wrong signal

The utilisation figure reported by nvidia-smi and by the standard DCGM field is the fraction of sampled time in which at least one kernel was executing. It is an occupancy-of-time measure, not a measure of how much work the device is doing. A single small kernel looping continuously reports one hundred per cent on a device that is almost idle in every meaningful sense.

For a batched inference server that is not an edge case, it is the normal state. Continuous batching keeps the device working essentially all the time whenever there is any traffic at all, so utilisation pins near its ceiling at low load and stays there. Between “comfortable” and “three hundred requests queued and users timing out” the metric does not move.

This is the general property to recognise: a saturating metric cannot drive a controller past its saturation point. The same objection applies to CPU percentage on a GPU service, where the number is mostly measuring your tokeniser. What GPU utilisation actually measures goes further into the counter itself.

What you actually care about is a latency target. So scale on something that is proportional to latency and unbounded above — the number of requests waiting to start.

Little’s law, worked

Little’s law says that for any stable queueing system, the mean number of items in the system equals the arrival rate multiplied by the mean time each item spends in it.

L = λ × W

  L = mean number of requests in the system (queued + in service)
  λ = mean arrival rate, requests per second
  W = mean time in system, seconds

Rearranged for what you want to control, with everything expressed per replica:

W = L / λ

Worked example. Labelled assumptions, all of them yours to replace:

  A1  one replica serves S = 4 concurrent generations at full speed
  A2  a generation takes  T = 6 s of service time at that concurrency
  A3  therefore one replica completes  μ = S / T = 4 / 6 = 0.67 req/s
  A4  your p95 latency budget is       W_max = 15 s
  A5  measured queue wait must then be W_q ≤ W_max − T = 15 − 6 = 9 s

Queue depth that corresponds to a 9 s wait, per replica:

  L_q = μ × W_q = 0.67 × 9 ≈ 6 requests waiting

So: target ≈ 6 queued requests per replica. Above that, the wait a new
arrival experiences exceeds the budget, and it exceeds it linearly —
12 queued is an 18 s wait, 24 queued is a 36 s wait.

Every term there is measurable in an afternoon. S and T come from a load test at fixed concurrency — load testing an AI endpoint is the harness. W_max is a product decision. Nothing in the derivation depends on a vendor, a model or a price, which is why it does not go stale.

The contrast with utilisation is now precise. Queue depth grows without bound as load exceeds capacity, and it is directly proportional to the thing users experience. Utilisation is bounded at 100 and stops carrying information long before the system is in trouble.

Choosing the target queue depth

One refinement before writing the manifest. If generations vary enormously in length — a 20-token classification behind a 2,000-token essay — mean service time understates the wait a short request experiences. Two options, and they compose:

• Measure queue wait directly instead of depth. Export the age of the oldest queued request, or a rolling p95 of wait time, and target that. It is the quantity you actually care about and it needs no assumption about service time.
• Separate the queues. Route long and short work to different pools with different targets. This is the same machinery as priority queueing and it solves head-of-line blocking at the same time.

If you export only one custom metric, export queue wait in seconds. Depth is a proxy for it, and the proxy is only as good as your service time estimate.

The autoscaler configuration

A HorizontalPodAutoscaler can scale on an external or custom metric, which is how the queue signal gets in. The metric must be published to the cluster by an adapter — the metrics pipeline differs by installation, so check what your cluster has before writing the metric.name. The shape below is stable across them.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-server
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-server
  minReplicas: 2
  maxReplicas: 12
  metrics:
    - type: Pods
      pods:
        metric:
          name: inference_queue_depth      # exported per pod
        target:
          type: AverageValue
          averageValue: "6"                # from the derivation above
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0        # react immediately
      policies:
        - type: Pods
          value: 4
          periodSeconds: 60                # at most +4 pods per minute
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 600      # 10 minutes of calm before shrinking
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120               # at most −1 pod per 2 minutes

The asymmetry between the two behaviours is the whole design. Scaling up late costs you queued requests; scaling down early costs you a cold start the moment traffic returns, and on GPU a cold start is expensive enough that the asymmetry should be extreme. Ten minutes of stabilisation before any scale-down is not conservative here, it is normal.

The metric must be exported per pod and must be the queue the pod owns. If every pod reports the depth of a shared external queue, the AverageValue target divides a global number by the replica count, which happens to work, but a per-pod internal queue is easier to reason about and localises the failure when one pod is slow.

Cold start decides your headroom

The reason GPU autoscaling feels different from web autoscaling is time-to-ready. Add the stages up for your own stack:

t_ready = t_provision + t_pull + t_load + t_warm

  t_provision  node exists and is Ready ................ 0 s if the node
                                                          is already in the
                                                          pool; minutes if
                                                          the cluster
                                                          autoscaler must add
                                                          one
  t_pull       image pulled ........................... 0 s if pre-pulled by a
                                                          DaemonSet
  t_load       weights read into host RAM and copied
               to device ............................... size ÷ effective
                                                          read throughput
  t_warm       first requests compile kernels, allocate
               the KV cache, populate caches ........... tens of seconds

Example, all assumptions labelled:
  40 GB of weights, read from a local NVMe cache at 2 GB/s → t_load ≈ 20 s
  same 40 GB pulled from object storage at 250 MB/s        → t_load ≈ 160 s
  node provisioning from a cold pool                       → t_provision 2–5 min

Now the headroom question answers itself. If t_ready is four minutes and your traffic can double in four minutes, the autoscaler cannot save you: by the time capacity arrives the queue has already blown the latency budget. You need standing headroom — spare replicas running — sized so that existing capacity absorbs whatever growth occurs within t_ready.

headroom_replicas ≈ ceil( peak growth rate (req/s per minute)
                          × t_ready (minutes)
                          ÷ per-replica capacity μ (req/s) )

With  growth 3 req/s per minute, t_ready 4 min, μ = 0.67 req/s:
  (3 × 4) / 0.67 ≈ 18 replicas of headroom.

That number is a decision, not a fate. Halving t_ready by pre-pulling the
image and caching weights on the node halves the headroom you must pay for
around the clock — which is usually the cheaper of the two fixes.

That is the practical takeaway of the whole page: on GPU, reducing time-to-ready is usually worth more than tuning the autoscaler, and getting weights warm before traffic is where most of that reduction lives.

Scaling to zero, and when not to

Scale-to-zero is attractive because idle GPU time is the largest controllable cost in most inference budgets. It is safe for batch-shaped and internal-tool-shaped workloads, where a first-request delay of a minute or two is an inconvenience rather than a failure.

It is unsafe for anything a user is waiting on synchronously, unless the arrival pattern has long, predictable gaps. The break-even arithmetic — idle hours paid for against cold starts incurred — is worked in full on serverless GPU, and it is the same calculation whether you implement scale-to-zero yourself or buy it.

If you do scale to zero, keep one thing warm on purpose: a small model or a remote API fallback that answers while the big pool spins up. Degrading to a cheaper answer beats a two-minute spinner, and graceful degradation covers what to tell the user while it happens.