Concurrency Planner
Little's law applied to LLM calls: arrival rate times time in flight gives the concurrent slots a workload needs, and what a limit you already have can sustain.
10.00 req/s × 10.70 s in flight = 107.00 average, × 1.50 safety factor.
- Generation time
- 10.00 s
- Total time in flight
- 10.70 s
- Average requests in flight (L = λW)
- 107.00
- Recommended slots
- 161
- Your limit
- 20
- Utilisation at your limit
- 535.00%
- Rate your limit sustains
- 1.87 req/s
- Headroom
- -87.00 slots
One equation, and why it is the one to memorise
L = λ × W — in-flight requests = arrival rate × time in flight
Little’s law needs no assumptions about arrival patterns, service distributions or scheduling policy. If a system is stable, the average number of requests inside it equals the rate they arrive multiplied by how long each one stays. That is all. It is the reason ten requests a second at eleven seconds each needs a hundred and ten concurrent slots, and the reason a concurrency limit of twenty makes that workload impossible no matter how much hardware sits behind it.
For LLM calls the W term is unusually large and unusually easy to estimate, because streaming duration is basically output length divided by token rate. A request that returns four hundred tokens at forty tokens a second occupies a slot for ten seconds — three orders of magnitude longer than a typical database query, which is why concurrency limits that were generous for a CRUD service are instantly binding for an AI feature. It also means the cheapest way to cut required concurrency is to shorten answers, not to speed anything up.
The safety factor is doing real work in the result. Arrivals cluster, some responses run long, and a pool sized exactly to the average will be full about half the time. What happens when it is full is the actual design question: requests queue, and queueing at high utilisation is where the latency you promised goes to die.