A KEDA ScaledObject for Queue-Driven Inference Pods

KEDA exists because the HorizontalPodAutoscaler cannot scale to zero and cannot read a queue. It solves both, and the way it solves the first is the thing to understand before writing the manifest.

Two controllers, one workload

A ScaledObject does not replace the HPA. When you create one, KEDA creates a HorizontalPodAutoscaler named keda-hpa-<scaledobject-name> by default and serves it metrics through the external metrics API. That HPA does all the scaling from minReplicaCount upwards, using the same ratio algorithm as any other HPA.

What KEDA keeps for itself is the transition between zero and one. The HPA cannot express it, so KEDA scales the Deployment to zero directly when the trigger reports inactivity, and back to one when it does not. This is why the zero boundary has its own threshold with its own name and its own semantics, and why tuning it like a normal target produces confusing behaviour.

The ScaledObject

For inference driven by a work queue — batch scoring, document processing, anything where a request can wait — the queue depth is both the scaling signal and the activity signal:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: batch-inference
  namespace: inference
spec:
  scaleTargetRef:
    name: batch-inference
  pollingInterval: 15
  cooldownPeriod: 600
  minReplicaCount: 0
  maxReplicaCount: 8
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: keda-trigger-auth-aws-credentials
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/123456789012/inference-jobs
        queueLength: "20"
        activationQueueLength: "1"
        awsRegion: eu-west-1
        scaleOnInFlight: "false"

Field by field, against the defaults KEDA documents: scaleTargetRef.kind defaults to Deployment and apiVersion to apps/v1, so both are omitted. pollingInterval defaults to 30 seconds and is shortened here because a GPU pod takes long enough to become ready that half a minute of detection latency is worth paying to remove. cooldownPeriod defaults to 300 seconds and applies only to the scale-to-zero decision. queueLength is the target messages per replica — the same role as averageValue on a plain HPA, defaulting to 5.

scaleOnInFlight defaults to true, meaning in-flight messages count toward the queue length. For inference that is usually wrong: messages a pod is already processing are not evidence that more pods are needed, and counting them keeps the autoscaler scaling up while the work is being done.

The activation threshold

activationQueueLength is the field that confuses people, and the two-controller split explains it. It is not a target; it is the boolean gate for zero. Above it, KEDA considers the trigger active and guarantees at least one replica. At or below it, and after the cooldown elapses, the workload goes to zero. It defaults to 0, so any message at all wakes the workload.

The distinction matters because the two numbers answer different questions. queueLength: 20 says “one replica per twenty messages”. activationQueueLength: 1 says “fewer than two messages is not worth a pod”. Set activation to 20 as well and a queue of fifteen messages sits unprocessed forever: not enough to activate, so no pod exists, so nothing drains it, so it never grows. That deadlock is the characteristic KEDA misconfiguration, and it produces no error anywhere — the ScaledObject reports Ready and the work simply never runs.

Timing, and the cost of being wrong

Scaling a GPU workload from zero has a longer critical path than a stateless web service, and every stage is serial: KEDA polls the queue, scales the Deployment to one, the pod goes Pending, a node may have to be provisioned, the image has to be pulled — and inference images with CUDA layers are large — then the model weights have to load into GPU memory before the pod is Ready.

Minutes, not seconds, and the last two stages are the ones you control. Shrinking the image is covered on reducing image size for model serving, and the probe configuration that stops Kubernetes killing the pod mid-load is on readiness probes for a slow-loading model. Set cooldownPeriod against that number rather than against the default: if a cold start costs four minutes, a 300-second cooldown means a gap of six minutes in the work stream costs you four minutes of latency to recover, and holding one replica warm may simply be cheaper.

What scale-in does to an in-flight request

Scaling up is forgiving; scaling down is where queue-driven inference loses work. Nothing in KEDA or in the HPA knows which pod is busy. The scale request goes to the Deployment, the ReplicaSet controller picks a victim from its own ordering, and the kubelet sends that container SIGTERM. A pod thirty seconds into a two-minute generation is as likely to be chosen as an idle one.

Three settings decide what that costs you. terminationGracePeriodSeconds on the pod is the window between SIGTERM and SIGKILL, thirty seconds by default, which is shorter than plenty of long-form generations — a streaming response cut at that boundary reaches the client as a truncated answer with no error. A preStop hook buys the endpoints controller time to remove the pod from its Service before the server stops accepting connections, which is what stops a scale-in producing a burst of connection resets. And the server itself has to handle SIGTERM by refusing new work while draining current work; a process that exits immediately on the signal makes the other two settings irrelevant.

You can influence which pod goes. The annotation controller.kubernetes.io/pod-deletion-cost on a pod in a ReplicaSet biases the ordering — lower cost is removed first — so a server that sets a low value when idle and a high one while processing steers scale-in towards the pods that have nothing to lose. It is a hint rather than a guarantee, and it is worth the plumbing only where a lost request is expensive.

For queue-driven work there is a cheaper answer: make the work replayable. If a pod dies mid-message and the message returns to the queue after its visibility timeout, scale-in costs latency instead of data. That means setting the queue’s visibility timeout longer than the worst-case inference, and only acknowledging a message after the result is durably written — at which point the grace period stops being a correctness control and becomes a tuning one. Priority queues for AI work covers the ordering side of the same design.

Verifying it

1. Apply the TriggerAuthentication first. For AWS the pod identity form is a short manifest with spec.podIdentity.provider: aws, and the KEDA operator’s service account needs permission to read the queue attributes. A ScaledObject with a broken auth ref reports a failure condition rather than silently scaling to zero.
2. Check the object: kubectl get scaledobject batch-inference -n inference. The READY and ACTIVE columns are the two states that matter — Ready means the trigger resolves, Active means it is currently above the activation threshold.
3. Confirm the generated HPA exists: kubectl get hpa keda-hpa-batch-inference -n inference. If it is missing, KEDA never reconciled and no scaling of any kind will happen.
4. Put one message on the queue and watch the Deployment go from zero to one. Then drain the queue and watch it return to zero after the cooldown, which is the half people forget to test.
5. Read the operator logs on failure: kubectl -n keda logs deploy/keda-operator. Trigger errors, including permission failures on the queue, surface there rather than on the ScaledObject.