Throughput vs Latency: You Can’t Optimise Both

A serving stack has one dial that matters and it points in two directions. Turn it up and the GPU produces more tokens per second in total while each individual user waits longer. There is no setting that is best at both, and the reason is arithmetic rather than engineering effort.

Say which one you mean

Three quantities get called “speed” and they are different:

A server can double the first while halving the second. Both statements are true simultaneously and neither is misleading; they are answers to different questions.

Why batching is nearly free, until it isn’t

From prefill vs decode: a decode step at batch size B reads the weights once and does B times the arithmetic, so its arithmetic intensity is about B FLOPs per byte. Step time is the larger of the two hardware costs:

t_step(B) ~= max( weight_bytes / bandwidth ,  B * flops_per_token / peak_flops )
                   \_______ constant _______/    \______ grows with B ______/

  memory-bound while  B < break_even_intensity  (~295 on an H100-class card)
  compute-bound after

Below the break-even point the second term is smaller, so t_step is essentially constant in B: doubling the batch doubles system throughput and costs each user nothing. This is the regime where batching looks like free money, and it is why continuous batching was such a large win.

Above it, step time grows linearly with B. Throughput has saturated — the chip is at its FLOP ceiling — while per-request rate is now falling in direct proportion to how many other people are in the batch. That is the trade, and the transition between the two regimes is a knee, not a gradual curve. Real systems blunt the knee (attention work grows with context, KV reads add bandwidth, kernels are imperfect), but the two asymptotes are exact and they are what the chart everyone draws is a picture of.

Little’s Law, and the wall at high utilisation

Batch size is only half the story, because a request that cannot get into the batch is waiting. Little’s Law relates the three quantities of any stable queueing system:

L = lambda * W        requests in system = arrival rate x time in system

and for the waiting part, at utilisation rho = lambda / capacity:

W_queue  ~  service_time * rho / (1 - rho)      (M/M/1 form)

  rho = 0.5  ->  1.0x service time waiting
  rho = 0.8  ->  4.0x
  rho = 0.9  ->  9.0x
  rho = 0.95 -> 19.0x
  rho = 0.99 -> 99.0x

Inference is not an M/M/1 queue and the constants will not match a real server, but the 1/(1−ρ) shape is robust across queueing models and it is the important part. An operator maximising throughput is maximising utilisation, and the last few percent of utilisation cost multiples of the whole service time in waiting. This is why the cheapest per-token capacity you can buy is also, reliably, the capacity with the worst tail latency: they are the same fact.

It also explains why latency degrades so suddenly. Between ρ = 0.8 and ρ = 0.95 the arrival rate rose by under 20% and the queue wait rose nearly fivefold. Nothing broke; you crossed a region where the derivative is steep.

The knobs, and who holds them

• Maximum batch size / maximum sequences. The primary dial. Operator’s.
• KV cache budget. Caps concurrency independently of the batch limit — see the cache formula. Operator’s, and often the real binding constraint.
• Chunked prefill size. Trades long prompts’ TTFT for everybody else’s smoothness. Operator’s.
• Target utilisation. The economic decision that sets where on the 1/(1−ρ) curve every user lives. Operator’s, and never published.
• Which endpoint the request goes to. Yours. This is the entire set of controls a caller has, and it is why routing across endpoints is the client-side answer to a server-side trade.

Ways to get some of both

The trade is real but it is not a single scalar, and most of the engineering in modern serving stacks is an attempt to buy back part of it rather than to sit somewhere on the line:

• Segregate the traffic. Two pools with different target utilisations — one lightly loaded for interactive requests, one run hot for background work — beats one pool at a compromise setting, because the compromise is bad for both populations. This is what a batch tier is, seen from the operator’s side.
• Disaggregate prefill and decode. The two phases want opposite things, so running them in separate pools lets each be scheduled for its own bottleneck instead of averaging the two.
• Reduce the work rather than reschedule it. Quantisation lowers the bytes decode must stream, prompt caching removes prefill entirely for a shared prefix, and speculative decoding converts idle arithmetic into latency at low batch. Each moves the whole curve rather than choosing a point on it.
• Admit less. Shedding or deferring work that has no deadline keeps utilisation off the steep part of the curve for the work that does. Counter-intuitively, refusing requests is a latency optimisation for the requests you accept.

What you are actually buying

When two providers serve the same weights at different prices, the difference is frequently a position on this curve rather than a difference in generosity. Cheap capacity is highly utilised capacity; highly utilised capacity has a long tail. A provisioned or dedicated deployment costs more per token precisely because you are paying for the idle headroom that keeps ρ low.

The practical consequence is that “which provider is fastest” is not a well-formed question without a load qualifier, and that a p50 comparison between a busy cheap endpoint and an idle expensive one will look far closer than the p99 comparison that your users will actually feel.