GPU Utilisation: Why the Number Is Lower Than You Think

The utilisation percentage on your dashboard does not mean what its name suggests. It can read 100% while the arithmetic units are essentially idle, and that is not a bug in the metric — it is a different metric from the one you want.

Three numbers, one word

Confusing the first for the third is the single most common misdiagnosis in a serving stack, and it sends people optimising kernels when the actual problem is that the batch is empty half the time.

The reason the first metric behaves this way is worth understanding rather than memorising. It is sampled: the driver periodically asks whether any kernel is executing and reports the fraction of samples that said yes. It has no view of how many of the device’s lanes that kernel occupies, how much of its bandwidth it uses, or whether it is doing useful work at all. A kernel that reads memory as fast as the device allows and a kernel that spins on a single lane are indistinguishable to it. That is not a defect in the counter — it is answering “is the device busy”, which is a legitimate question and a different one from “is the device being used well”.

Why MFU is low even when nothing is wrong

The bound comes straight from arithmetic intensity. At batch B and b bytes per parameter, decode performs about 2B/b FLOPs per byte read. The hardware’s ridge point is peak FLOP/s divided by peak bandwidth. So:

MFU_ceiling  ~=  intensity / ridge_point

Example: intensity at B=1, bf16   = 1 FLOP/byte
         ridge = 1e15 / 3e12      = 333 FLOP/byte

MFU_ceiling = 1 / 333 = 0.3%

A single-stream decode that achieves 0.3% of peak FLOPs is running at the ceiling. No kernel rewrite improves it, because the weights still have to be read. The way to raise MFU is to raise intensity, and the way to raise intensity is to batch. Reported alongside that, memory bandwidth utilisation near peak is the success condition — and it is the metric to put on the dashboard.

Prefill is the opposite and should be judged differently. With hundreds or thousands of tokens sharing each weight read, prefill sits well above the ridge and a low MFU there genuinely does indicate a problem: poor kernel selection, an unhelpful sequence length, or excessive padding.

Where the real bubbles are

Empty or ragged batches

With static batching, a batch is formed, run to completion, and only then replaced. Since sequences finish at different lengths, the slots of finished sequences sit idle until the longest one completes. Continuous batching — admitting a new sequence into a freed slot at the next step — is the fix, and it is the single largest throughput feature in a modern serving engine.

Prefill blocking decode

A long prompt arriving mid-stream occupies the device for a compute-heavy prefill while every decoding sequence waits. The symptom is inter-token latency spiking for unrelated users. Chunked prefill splits the prompt into pieces and interleaves them with decode steps, trading a slightly slower prefill for a much steadier decode.

Host-side serialisation

Tokenisation, sampling logic, JSON schema constraint checking, logging and Python overhead all run between kernel launches. At small batch the per-step device work can be a millisecond or two, which is the same order as the host work, so the device waits. Graph capture and batched sampling exist for this reason.

Collective waits

Under tensor parallelism every layer ends in a collective. If the devices are not perfectly balanced — different clock throttling, uneven memory pressure — the fastest device idles at each barrier, and that idle time appears as high occupancy and low throughput.

A diagnostic order of operations

• Establish which regime you are in. If output is being generated, decode dominates and low MFU is expected. Judge on bandwidth utilisation and on tokens per second across the whole batch.
• Look at the batch size actually achieved, per step, over time. If your engine reports running versus waiting sequences, a large waiting queue alongside a small running batch means memory is the limit and the KV cache is the thing to shrink.
• Compare inter-token latency distribution, not the mean. Steady decode with periodic spikes is prefill interference. Uniformly slow decode is the bandwidth bound and is not a bug.
• Check the host. If device work per step is comparable to the gap between steps, the bottleneck is not on the device at all.
• Only then look at kernels. They are the last place to look and the first place people look.

What to actually target

Utilisation is not a goal; it is an intermediate. The goals are throughput at an acceptable latency, and cost per token. A stack that runs at 60% occupancy while meeting a latency target at a batch size set deliberately from the memory budget is healthier than one pinned at 100% because a queue is permanently full.

The pairing worth putting on one chart is tokens per second across the batch against p95 inter-token latency, plotted as batch size rises. It shows the free region below the ridge, the knee where per-user latency starts to degrade, and the point where memory runs out — the three facts that actually govern a serving deployment.

One consequence for capacity planning follows from all of this and is easy to miss. Because the free region exists, an endpoint serving a handful of users is using a small fraction of the hardware it occupies, and adding users to it costs almost nothing until the knee. So the efficient move when utilisation is low is rarely to buy a smaller device — it is to consolidate more traffic onto the one you have. The same arithmetic explains why a shared endpoint operated across many tenants is structurally cheaper per token than a dedicated one at low volume, and why that advantage disappears once you can fill a batch yourself.