GPU Sizing for Concurrent Users

Concurrent requests, context length and a target token rate into a GPU count — with the memory limit and the bandwidth limit computed separately so you can see which one binds.

Concurrent requests at peakRequests generating at the same instant, not registered users.

Target rate per userTokens per second each stream should sustain. Faster than reading speed is usually wasted.

tok/s

Average resident contextTokens held per sequence, prompt plus generation so far.

tokens

ParametersIn billions.

Weight precisionBits per parameter as served.

bit

VRAM per GPUYour own hardware's figure.

Memory bandwidth per GPUFrom your card's specification — a placeholder until you replace it.

GB/s

Bandwidth efficiencyAchieved share of peak. Measure it; it scales everything below.

Layersconfig.json → num_hidden_layers

KV headsnum_key_value_heads. With grouped-query attention this is 4–8× smaller than the attention head count — use the KV one.

Head dimensionhidden_size ÷ num_attention_heads

KV cache precisionBits per element. 16 unless your engine quantises the cache (8 and 4 are common options).

bit

Runtime overheadShare of weights + KV reserved by the engine.

GPUs needed

65 concurrent sequences per GPU, limited by memory. At that batch each stream runs at about 27.6 tok/s.

Weights on every GPU: 7.45 GB
KV cache per token: 128.0 KB
KV cache per sequence: 0.98 GB
Memory-bound seats per GPU: 65
Bandwidth-bound seats per GPU: 92
Binding constraint: memory
Seats per GPU: 65
Per-stream rate at that batch: 27.6 tok/s
Aggregate output per GPU: 1792 tok/s
GPUs for the stated peak: 1

What this assumes: One GPU holds the whole model — tensor parallelism, which splits the weights and adds bandwidth, is not modelled and changes both limits. Decode only: prefill contends for the same GPU, so treat the seat count as an optimistic ceiling and leave room for it. The bandwidth limit assumes decoding is memory-bound, which stops being true at very large batches where compute takes over. Every sequence is assumed to sit at the average context; a real mix has a tail that reserves more.

Two limits, and the one that binds is not always memory

memory: (W + B·K) × (1 + overhead) ≤ VRAM
speed: bandwidth ÷ (W + B·K) ≥ target tokens/s

Batching an LLM server is unusually forgiving in one respect: a decode step reads the weights once no matter how many sequences are in the batch, so the second concurrent user is nearly free in bandwidth terms. That is why continuous batching works at all. What is not free is each sequence’s KV cache, which must be both stored and re-read every single step. Both limits above are the same expression — weights plus batch times cache — measured once against capacity and once against time.

Reading which one binds is the useful output. If memory binds, you are leaving speed on the table and the fixes are memory fixes: quantise the KV cache, cap the context, or take a card with more VRAM. If bandwidth binds you have spare memory you cannot use, and the fix is a smaller or more heavily quantised model, or accepting a lower per-user rate. A target of twenty tokens a second is already faster than most people read; halving it can double the seats per GPU.

One failure mode has no fix through scale. If a single sequence cannot hit the target rate, more GPUs will not help — every one of them is the same speed, and per-user latency is a property of one GPU’s bandwidth against one model’s weights. Adding hardware buys throughput, never latency.

GPU Sizing for Concurrent Users

Two limits, and the one that binds is not always memory

The reasoning behind this

Related tools