Backpressure and Queue Depth in AI Pipelines

A queue does not absorb overload. It converts a fast failure into a slow one, and if it is unbounded it converts a slow failure into a total one. Backpressure is the design of what you refuse, and refusing early is nearly always better than refusing late.

The queue becomes the outage

Here is the sequence, and it is the same every time. Arrival rate exceeds service rate for a while. The queue grows. Latency grows with it, because latency is queue time plus service time. Clients hit their own timeouts and retry, which increases the arrival rate. The queue grows faster. By the time you look, the queue contains twenty minutes of work, every item in it is past its deadline, and the system is spending all of its capacity producing answers that will be discarded on arrival.

The dependency’s properties make each stage worse. Service time is seconds, so a queue drains slowly. Every item is billed, so working through a backlog of dead requests is not just wasted capacity but money spent on output nobody receives. And retries are expensive rather than free, so the amplification loop has a price attached.

The starting point is therefore not “how big should the queue be” but “the queue must be bounded, and I must decide what happens at the bound”. An unbounded queue is not a capacity decision deferred; it is a capacity decision made badly.

Depth is the wrong signal; wait is the right one

“Reject when the queue exceeds 500” is a threshold with no meaning, because 500 items is thirty seconds of work on a good day and fifteen minutes on a bad one — and a bad day is precisely when the rule fires. Convert depth into time before you make a decision:

// Little's Law, used as an admission control rule.
//   estimated wait = queue depth / service rate
// where service rate = concurrency / average service time.

function estimatedWaitMs(depth: number, concurrency: number, avgServiceMs: number) {
  const perSecond = concurrency / (avgServiceMs / 1000);
  return (depth / perSecond) * 1000;
}

function admit(job: Job, q: QueueStats): "run" | "reject" {
  const wait = estimatedWaitMs(q.depth, q.concurrency, q.p50ServiceMs);
  // Reject anything that cannot be finished inside the caller's deadline.
  // Accepting it would mean paying for an answer that arrives after
  // everyone has stopped listening.
  if (wait + q.p95ServiceMs > job.deadline.remaining()) return "reject";
  return "run";
}

Two properties make this better than a depth threshold. It adapts automatically when service time changes, which is the situation that breaks fixed thresholds. And it produces an honest rejection message: you can tell the caller the estimated wait and the reason, which lets a well-behaved client back off intelligently instead of retrying immediately into the same wall.

The general principle is worth stating on its own. Never start work you cannot finish in time. In a system with a free dependency that rule is about efficiency. Here it is about money.

What to shed, in order

Load shedding is easier if the priority classes were decided in advance, in calm conditions, rather than during the incident. A workable default set, shed from the top:

• Speculative and prefetch work. Suggestions nobody asked for, precomputed summaries, background enrichment that could run tomorrow. Free to drop, and often a surprising share of volume.
• Retries of already-failed work. Under overload, retries are the amplifier. A retry budget expressed as a fraction of total requests — the client-side retry budget described in Google’s SRE book — caps this structurally instead of relying on each caller to be polite.
• Requests whose deadline has already passed. Free capacity, zero user impact, and you would be amazed how much of a saturated queue this is.
• Low-tier or over-quota tenants. If you have tiers, this is what they are for. Shedding uniformly during overload means your most valuable traffic suffers exactly as much as your least.
• Interactive requests from waiting users. Last. When you are shedding these, the answer is capacity, not policy.

Whatever you shed, shed it with a 429 or 503 and a Retry-After. A rejection that carries no timing information invites an immediate retry, and an immediate retry is the thing you were trying to prevent.

The counterintuitive one: LIFO under overload

A queue normally serves oldest-first, which is fair. Under sustained overload, fairness is the wrong objective: the oldest item is the one most likely to have already timed out, so first-in-first-out means systematically serving the requests least likely to be wanted.

Serving newest-first during overload means the items you complete are the ones whose callers are still waiting, so some requests succeed instead of all of them failing slowly. The old items are then dropped on expiry rather than executed. This is a known technique in overload-control literature and it is genuinely uncomfortable, because it abandons the oldest work deliberately — but the alternative under overload is not “everyone gets served eventually”, it is “everyone gets served too late”.

If that feels too aggressive, the milder version gets most of the benefit: keep FIFO ordering, but check each item’s deadline at dequeue time and drop the expired ones without executing them. That alone converts a queue full of dead work into a queue that drains at the rate of live work.

Shedding on budget, not just on load

This dependency adds an axis that classical backpressure does not have. Load is not the only thing that can be exhausted; so can money. A runaway loop, a viral page or a bug that retries forever can be entirely within your concurrency limits and still spend a month of budget in an afternoon.

So run a spend limiter alongside the load limiter, with the same structure: a rolling window, a threshold, and a shedding policy tied to the same priority classes. When the hourly spend rate exceeds its ceiling, shed speculative work first, then downgrade to a cheaper rung of the degradation ladder, and only then reject. Alert on the rate of spend, not the total — a total tells you after the fact, a rate tells you within minutes.

Per-tenant budgets belong in the same mechanism. Without them, one customer’s runaway integration is served out of everyone else’s capacity and everyone else’s budget, and the first signal is the invoice.

Pushing backpressure to the producer

Shedding at the queue is the last line. It is better for the producer to slow down, and the mechanisms for that are unglamorous and effective: bound the queue so a synchronous producer blocks naturally; return 429 with a Retry-After to remote producers so a well-behaved client throttles itself; expose the current estimated wait in the response so a client can decide whether to submit at all; and, for internal producers, let the semaphore in front of the dependency be the thing that blocks, since a producer awaiting a permit is a producer applying its own backpressure for free.

The one thing that never works is asking producers to be considerate. Backpressure has to be a mechanism, because politeness does not survive an incident and it does not survive a client you do not control.