Load Balancing Across Multiple Inference Endpoints

Load balancers were designed for requests that are alike: short, cheap, and roughly equal. Model calls are none of those. One request may occupy a backend for two hundred milliseconds and the next for ninety seconds, which breaks the assumption every classic algorithm rests on.

Why round-robin is wrong here

Round-robin equalises request counts, on the assumption that equal counts imply equal load. For generation, request cost varies by two or three orders of magnitude — a yes/no classification and a 10,000-token document summary arrive through the same endpoint — so equal counts imply nothing at all. A backend can be handed its fair share of requests and be four times as loaded as its neighbour.

Least-connections is a large improvement precisely because outstanding requests are a decent proxy for occupancy: a backend still working on three long generations shows three connections and is correctly avoided. It is the right default, and the strategies below are refinements of it rather than alternatives to it.

The strategies, by what they optimise

Power of two choices, with code

Picking the globally best backend is a bad idea for a reason that is not obvious: every router makes the same choice at the same moment, so the “best” backend is instantly overwhelmed and the metric that named it is stale by the time the traffic lands. The standard fix is to sample two backends at random and take the better of the two — the power-of-two-choices result, which gets most of the benefit of global knowledge with none of the herding.

type Backend = {
  id: string;
  inFlight: number;
  ewmaMs: number;          // decaying mean of recent time-to-first-token
  costPerMTok: number;
  openUntil: number;       // circuit breaker: 0 when closed
};

const ALPHA = 0.2;         // EWMA weight on the newest sample

export function observe(b: Backend, ttftMs: number) {
  b.ewmaMs = b.ewmaMs === 0 ? ttftMs : ALPHA * ttftMs + (1 - ALPHA) * b.ewmaMs;
}

/** Cost of sending one more request here. Lower is better. */
function score(b: Backend, costWeightPerMs: number): number {
  // inFlight+1 is the queue this request would join; multiplying by the
  // observed service time estimates the wait rather than the count.
  const predictedWaitMs = (b.inFlight + 1) * b.ewmaMs;
  return predictedWaitMs + b.costPerMTok / costWeightPerMs;
}

export function pick(pool: Backend[], costWeightPerMs = Infinity): Backend | null {
  const live = pool.filter((b) => b.openUntil < Date.now());
  if (live.length === 0) return null;      // everything is broken: shed, do not spin
  if (live.length === 1) return live[0];

  // Two independent samples, then the better of the two. Randomness is the
  // point -- it is what stops every router in the fleet choosing identically.
  const i = Math.floor(Math.random() * live.length);
  let j = Math.floor(Math.random() * (live.length - 1));
  if (j >= i) j++;

  const a = live[i], b = live[j];
  return score(a, costWeightPerMs) <= score(b, costWeightPerMs) ? a : b;
}

Two design notes. The score multiplies queue depth by observed service time rather than using either alone, because a backend with four in-flight requests and a 200 ms service time is a better bet than one with two in-flight and a 3-second service time. And costWeightPerMs is an explicit exchange rate between money and milliseconds; setting it to infinity gives you pure latency routing, and setting it low gives you cost routing that still refuses to pile onto a stalled endpoint. Making that exchange rate a parameter beats arguing about which objective is correct.

The tension nobody mentions: cache affinity

Here is the conflict that catches people. Prefix caches live on specific serving instances. If your requests share a long system prompt, sending them to whichever backend is fastest right now is actively destroying the hit rate you built the prompt structure to get — and losing a cache hit can cost far more, in both money and time to first token, than the routing decision saved.

The resolution is a two-tier policy rather than one rule. Prefer the backend that holds your prefix; abandon it only when it is clearly unhealthy:

const sticky = pool.find((b) => b.id === affinityFor(prefixHash));

if (sticky && sticky.openUntil < Date.now()
           && sticky.ewmaMs < 2.5 * medianEwma(pool)) {
  use(sticky);              // keep the cache; tolerate being somewhat slower
} else {
  use(pick(pool));          // it is genuinely bad: rebalance and rewarm
}

The 2.5× is a policy choice, not a discovered constant — it encodes how much latency you are willing to pay to preserve a cache hit, and the right value depends on how large your cached prefix is relative to the rest of the request. Write it down as a named constant rather than leaving it implicit in a comparison.

It is also worth being clear that balancing and failover are different products of the same machinery, and most teams want the second while building the first. Failover is an ordered list: use the primary until it is unhealthy, then the next. It preserves cache affinity, keeps behaviour predictable, and makes billing legible. Balancing spreads traffic continuously to raise aggregate capacity and smooth tails, at the cost of all three. If your motivation is “I do not want an outage to take me down”, you want failover with health checks and you should not spread traffic at all. If it is “one endpoint cannot serve my volume”, you want balancing. Building the second when you needed the first is how a system ends up with an unexplainable output distribution and a cache hit rate of nearly zero.

Health, in either case, has to be defined before it can be checked. A synthetic probe every few seconds tells you an endpoint is reachable and nothing about whether it is admitting your requests; the useful signal is passive, taken from real traffic — success rate, time to first token, and 429 rate over a short sliding window, with a minimum request count before the window is allowed to open a breaker at all. Otherwise a single failure on a low-traffic endpoint takes it out of rotation, and it never gets the traffic it would need to prove itself healthy again.

Failure modes to design against

• The herd. All routers pick the same newly-fast backend, overload it, then all flee to the next one. Latency-aware routing without randomisation oscillates by construction. Power-of-two-choices and EWMA damping are the standard treatments.
• The fast failure trap. A backend returning instant errors looks like the fastest one to any latency-based score. Score only successful requests, and gate on a circuit breaker — this is the classic way a latency-aware balancer routes 100% of traffic into a broken host.
• Cold starts as false signal. A scaled-to-zero backend’s first request is enormously slow and says nothing about its steady state; see cold starts. Exclude, or heavily discount, the first sample after an idle period.
• Retry storms across the pool. A failed request retried against every backend in turn multiplies load during exactly the incident where load is the problem. Retries must draw from the same budget as first attempts.
• Silent quality drift. Two endpoints serving the same model name may not be serving the same artefact — different precision, different builds. Balance across them and your output distribution becomes a mixture you never chose; the probe in detecting a changed endpoint is worth running per backend rather than per model name.