Batching and Latency Simulator: The Trade, Live
Throughput and per-request latency across every batch size, from two step-time anchors you measure yourself, with the queue wait that makes high utilisation expensive.
11.1× the single-request rate, bought with a per-request decode of 6895 ms instead of 4800 ms. At 1.20 req/s that is 52% utilised, and the queue adds 462 ms.
- Decode step time at this batch
- 17.2 ms
- Server throughput
- 928 tok/s
- Per-request decode time
- 6895 ms
- Requests completed per second
- 2.32
- Arrival rate
- 1.20 req/s
- Utilisation ρ
- 51.7%
- Expected queue wait
- 462 ms
- End-to-end, including prefill
- 7577 ms
You cannot have both, and the curve shows why
Batching is the reason inference is affordable. Weights have to be read from memory for every decode step regardless of how many sequences are in flight, so serving sixteen requests together reads them once instead of sixteen times, and throughput rises almost linearly until arithmetic rather than bandwidth becomes the limit. That is the solid line.
The dashed line is what it costs. A bigger batch makes every step slower for everyone in it, so each individual request takes longer to finish than it would have alone. Nobody in the batch is being starved; they are all being served at once, slightly worse. This is why the same request takes 400 ms at three in the morning and four seconds at peak on an unchanged deployment, and why a p99 latency target and a cost-per-token target pull in opposite directions.
The queueing row is the part people leave out, and it is the part that bites. Wait time does not rise smoothly with load — it rises as ρ/(1−ρ), which is nearly flat until utilisation passes about 70% and then goes vertical. Sizing a fleet to run at 90% utilisation looks efficient on a spreadsheet and produces a latency distribution with a tail you cannot explain to anyone. Set the arrival rate near capacity and watch the wait column outgrow every other number on the page.
Two anchors decide everything here and neither can be guessed: step time at batch 1 and at your maximum batch, on your hardware, with your model. They take ten minutes to measure with a load generator, and every number above is exact arithmetic on top of them.