Throughput Planner

Target request rate, prompt and output lengths into required capacity, by converting each request into GPU-seconds of prefill and decode.

Average request rateRequests per second, averaged over a busy hour.

req/s

Peak-to-average factorBusiest minute ÷ average. Size for the peak, or the peak becomes a queue.

Input tokens per requestSystem prompt, tools, retrieved context and the user turn.

tokens

Output tokens per requestIncluding reasoning tokens if the model emits them.

tokens

Prefill throughput per GPUAggregate input tokens per second one GPU processes under load. Benchmark your own engine — this default is a round placeholder.

tok/s

Decode throughput per GPUAggregate output tokens per second across the whole batch, not per stream. Also yours to measure.

tok/s

Target utilisationNever plan for 100%: queueing delay climbs steeply above roughly 80%.

GPUs required

0.472 GPU-seconds per request × 10.00 req/s at peak = 4.72 GPUs of work, run at 70% utilisation.

Prefill time per request: 222 ms
Decode time per request: 250 ms
GPU-seconds per request: 0.472
Peak request rate: 10.00 req/s
Tokens/s at peak: 24,000 tok/s
GPU-equivalents of work at peak: 4.72
GPUs at target utilisation: 7
Actual utilisation at that count: 67.46%
Sustainable average rate: 5.19 req/s

What this assumes: Prefill and decode are treated as one pool of GPU-seconds, which is what continuous batching approximates; a deployment that separates prefill and decode onto different machines has to be sized as two pools. Both throughput figures are yours to measure at your own batch size and context length — they are not constants, and prefill in particular falls as context grows. Utilisation is an average: this says nothing about the wait a request sees, which is the queue planner’s job.

Capacity is a time budget, not a token budget

GPU-seconds per request = input ÷ prefill rate + output ÷ decode rate
GPUs = ⌈ (requests/s × peak factor × GPU-seconds) ÷ target utilisation ⌉

The instinct is to add up tokens per second and divide, but input and output tokens are not the same unit of work. Prefill processes the whole prompt in parallel and is compute-bound, so a GPU chews through thousands of input tokens a second. Decode emits one token per step per sequence and is bandwidth-bound, and its aggregate rate is far lower. Converting both into seconds of GPU time first is what makes them addable, and it is why a retrieval-heavy workload with short answers sizes completely differently from a long-generation one.

Two multipliers do most of the damage in real plans. The first is the peak factor: traffic that averages five requests a second does not arrive five per second, and a fleet sized for the average is permanently behind during the busy hour. The second is target utilisation. Queueing delay is not linear in load — it rises gently to about 70%, then sharply, and at 95% a small burst produces a wait nobody budgeted for. Planning to 70% is not waste; it is the cost of predictable latency.

What this deliberately does not do is guess your throughput numbers. Measure them on your engine, your quantisation, your batch size and your real context length, then come back — the answer is only as good as those two fields.

Throughput Planner

Capacity is a time budget, not a token budget

The reasoning behind this

Related tools