Queue Depth and Wait Time Estimator
Arrival rate, service time and worker count into the expected queueing delay, the chance of waiting at all, and the workers needed to hold a wait target.
Utilisation 80.00%. 8.70% of arrivals find every worker busy and have to queue.
- Offered load (λ × S)
- 40.00 erlangs
- Utilisation
- 80.00%
- Probability of queueing (Erlang C)
- 8.70%
- Average wait, all arrivals
- 87 ms
- Average wait, those who queue
- 1.00 s
- Average total time in system
- 10.09 s
- Chance of waiting over 2.00 s
- 1.18%
- 95th percentile wait
- 553 ms
- Workers to hold the wait under 2.00 s
- 43
Why the wait explodes rather than climbs
offered load a = λ × S
utilisation ρ = a ÷ c
average wait Wq = C(c, a) × S ÷ (c − a)
The offered load a is how many workers would be busy on average if there were always one free — arrival rate times service time, a pure number called erlangs. Divide by the workers you have and you get utilisation. The wait is what falls out of the gap between the two, and the shape of that expression is the whole lesson: the divisor is c − a, so as capacity approaches load the wait does not rise steadily, it goes to infinity.
That is why a system at 70% utilisation feels fine and the same system at 95% feels broken, without anything having changed except a little more traffic. It is also why adding one worker to a saturated pool can have an effect out of all proportion to its size, and why capacity plans that target high utilisation are betting on traffic being smoother than traffic is.
For LLM work the service time is long — seconds, not milliseconds — and long service times make every queue worse at the same utilisation, because one unlucky arrival waits behind whole responses rather than behind quick queries. The practical response is usually not more capacity: it is separating interactive traffic from batch traffic so a background job cannot park itself in front of a person, and shedding or deferring work honestly when the queue is deep, rather than letting everyone wait and time out together.