A Horizontal Pod Autoscaler on a Custom Inference Metric

A model server pinned at 100% GPU is often perfectly healthy, and a model server at 20% CPU can have fifty requests queued. Scaling inference on CPU utilisation measures the wrong component of the machine.

Why CPU is the wrong signal for a GPU pod

In a typical inference server the CPU does tokenisation, HTTP handling and scheduling while the GPU does the arithmetic. Under heavy load the process is blocked waiting on the GPU, so CPU utilisation can fall as latency rises — the metric moves the wrong way at exactly the moment you need to scale.

GPU utilisation is not much better as a scaling signal, because a well-batched server should sit near 100% whenever it has work. It tells you the accelerator is busy; it does not tell you whether requests are waiting. The number that does is queue depth: how many requests have arrived and not yet started. vLLM, for instance, exposes vllm:num_requests_waiting and vllm:num_requests_running on its Prometheus endpoint, and the first of those is a direct measure of unserved demand.

The four pieces of the pipeline

The HPA cannot read Prometheus. It reads the Kubernetes metrics APIs, and something has to put your number there:

• The server exposes the metric on an HTTP endpoint in Prometheus text format.
• Prometheus scrapes it and stores it with labels identifying the pod and namespace. Those labels are what make the metric attributable to a workload.
• An adapter serves it through the aggregated API custom.metrics.k8s.io for pod-attached metrics, or external.metrics.k8s.io for metrics with no Kubernetes object behind them. The Kubernetes documentation is explicit that these APIs require a separate implementation; nothing in a default cluster serves them.
• The HPA queries the API on its sync interval, which the Kubernetes documentation gives as 15 seconds by default via --horizontal-pod-autoscaler-sync-period.

Every one of those can fail independently, so build it in that order and verify each before adding the next.

Exposing the metric to the adapter

Using the Prometheus Adapter, a rule tells it which series to expose, how to associate it with Kubernetes objects and what query to run:

rules:
  - seriesQuery: 'vllm:num_requests_waiting{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: { resource: "namespace" }
        pod: { resource: "pod" }
    name:
      matches: "^vllm:(.*)$"
      as: "${1}"
    metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

The resources.overrides block is the part that matters and the part most often wrong. It maps Prometheus labels onto Kubernetes resources, and without it the adapter serves a metric that no HPA can associate with any pod. The name block renames the series into the metric name the HPA will use — here stripping the vllm: prefix to give num_requests_waiting, because a colon is awkward in a manifest.

Verify the adapter before touching the HPA. The API is queryable directly:

kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/inference/pods/*/num_requests_waiting" \
  | jq .

A list of pods each with a value is success. A 404 means the rule did not match or the adapter is not registered as an APIService; an empty list means the series exists but the resource association failed.

The HorizontalPodAutoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference
  minReplicas: 2
  maxReplicas: 12
  metrics:
    - type: Pods
      pods:
        metric:
          name: num_requests_waiting
        target:
          type: AverageValue
          averageValue: "4"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120

Three choices in that manifest are specific to GPU workloads. The metric source is Pods with target type AverageValue, because queue depth is a per-pod count and Utilization is only available for resource metrics. The scale-up policy is capped at two pods a minute, because each new pod may need a new GPU node and requesting twelve at once produces a stampede of node requests that will not be satisfied. And the scale-down stabilisation window is lengthened well past the 300-second default the Kubernetes documentation gives, because tearing down a GPU pod that took several minutes to load its weights is expensive to undo.

When the metric disappears

A custom-metric HPA has more ways to be silently inert than a CPU one, because four components have to keep agreeing. The status object is where that shows up, and it is worth reading before touching any numbers:

kubectl describe hpa inference -n inference

Three conditions appear there. AbleToScale covers whether the HPA can fetch and update the target’s scale at all. ScalingActive is the one that goes False when metrics cannot be retrieved, with an accompanying FailedGetPodsMetric event naming the metric it could not fetch. And ScalingLimited True with reason TooManyReplicas means the HPA computed a higher count and was capped by maxReplicas— the autoscaler working correctly and telling you the ceiling is the problem.

The important behaviour is what happens while ScalingActive is False: the HPA holds the replica count where it is. It does not fall back to CPU, does not scale to minReplicas, and does not scale up. A metrics pipeline that breaks during quiet hours therefore freezes the fleet at its overnight size and it stays there through the morning peak, with the Deployment looking healthy the whole time. Alert on the ScalingActive condition rather than on replica count.

Two subtler versions of the same problem. Pods that are not yet Ready are treated specially — Kubernetes documents an initial readiness delay of 30 seconds by default via --horizontal-pod-autoscaler-initial-readiness-delay — which matters here because a GPU pod can be un-Ready for minutes while it loads weights, so the metric average during a scale-up is computed over fewer pods than you think. And a queue-depth metric legitimately reads zero when there is no traffic, which drives the HPA straight to minReplicas; that value, not the target, is what decides how much warm capacity survives a quiet period, and on a workload with slow starts it should be set from your worst acceptable cold start rather than from tidiness.

Choosing the target value

The HPA algorithm is published, and it is worth working with rather than guessing at. Kubernetes documents the desired replica count as the ceiling of current replicas multiplied by the ratio of the current metric value to the target value, with scaling skipped while that ratio is within a tolerance of 1.0 — documented as 0.1 by default.

Read that as: the target is the queue depth per pod you are willing to run at in the steady state. If four replicas are averaging eight waiting requests against a target of four, the ratio is 2.0 and the HPA asks for eight replicas. If your acceptable queue is one, the same load asks for thirty-two. So the target is a latency decision expressed as a number, and it follows from your batch size: a server configured for a maximum batch of eight is not under pressure at a queue of four, because those requests will join the next batch. A server with a maximum batch of two is badly behind at the same number.

The tolerance has a practical consequence too. A target of 1 means a measured value of 1.1 is within tolerance and nothing happens; small targets make the autoscaler coarse. Prefer a target large enough that a 10% band is narrower than the change you care about.

The HPA has one limit that no configuration removes: it will not scale a Deployment to zero. If the workload should have no pods at all when idle, a KEDA ScaledObject is the mechanism, and it drives an HPA underneath for everything above one replica. Replica count is also only half the problem — new replicas need nodes to land on, which is the cluster autoscaler’s job.