Memory Bandwidth Is the Real Bottleneck

Two devices, one with twice the peak FLOPs and the same memory bandwidth, will generate text at almost exactly the same speed for a single user. This is not a driver problem. It is the consequence of a ratio you can calculate in one line.

The ceiling, in one line

Producing one token requires reading the weights that token uses. For a dense model, all of them:

t_token  >=  (P * b) / BW

P  = parameters
b  = bytes per parameter
BW = achievable memory bandwidth in bytes/second

so   tokens_per_second  <=  BW / (P * b)

Two caveats keep this honest. BW is achievable, not the datasheet peak — real kernels typically realise a large fraction of peak, not all of it, and the fraction depends on access patterns. And the KV cache is read too, so the true denominator is slightly larger than P × b, by an amount that grows with context length. Both push the real number below the ceiling. Neither lets you exceed it.

Sanity-check it against something you have seen: a 70B model at fp16 is 140 GB of weights. On a device advertising 3 TB/s of high-bandwidth memory — an order of magnitude vendors have quoted on datasheets since around 2024, used here purely as an input — the ceiling is 3000/140 ≈ 21 tokens/second. That is close to what single-stream generation of a model that size feels like, and it was derived without measuring anything.

Arithmetic intensity

Arithmetic intensity is FLOPs performed per byte moved from memory. It is the single number that decides which resource binds.

Using a weight once costs one multiply and one add: 2 FLOPs per parameter per token. Loading that weight costs b bytes. At batch size B, the same load serves B tokens, so:

intensity = (2 * P * B) / (P * b) = 2B / b   FLOPs per byte

at b = 2 (bf16):  intensity  ~=  B

So the arithmetic intensity of decode is, to a very good approximation, the batch size. At B = 1 the hardware performs about one floating-point operation for every byte it reads. Prefill is the same expression with the prompt length in place of the batch, because a prompt of S tokens also reuses each loaded weight S times — which is the entire reason prefill is compute-bound and decode is not.

The roofline and its ridge point

The roofline model says attainable performance is the lesser of what the arithmetic units can do and what the memory system can feed:

attainable_FLOPS = min( peak_FLOPS , BW * intensity )

ridge_point = peak_FLOPS / BW      (FLOPs per byte)

The ridge point is where the two lines cross: below it you are memory-bound, above it compute-bound. Take a device with 1,000 TFLOP/s of dense low-precision throughput and 3 TB/s of bandwidth — again, plausible orders of magnitude used as inputs, not a claim about any product. The ridge point is 1e15 / 3e12 ≈ 333 FLOPs per byte.

Decode at B = 1 sits at an intensity of about 1. The ridge is at 333. You are two and a half orders of magnitude away from being able to use the arithmetic units, which means roughly 0.3% of peak FLOPs is not a misconfiguration — it is the arithmetic working correctly.

Prefill lands on the other side of the same ridge. A 2,000-token prompt reuses each loaded weight two thousand times, giving an intensity of roughly 1,000 FLOPs per byte at bf16, comfortably past 333. So a single request moves across the ridge point in the middle of its own lifetime: it starts compute-bound while reading your prompt and becomes memory-bound the moment it starts writing an answer. That is the mechanical reason time to first token and inter-token latency respond to completely different interventions, and the reason providers price input and output tokens differently rather than as a margin decision.

Batching moves the bottleneck

Set intensity equal to the ridge point and solve: with intensity ≈ B and a ridge of 333, decode becomes compute-bound somewhere around a batch of 333 concurrent sequences. That single number explains most of how inference is served commercially.

• Throughput scales nearly free with batch, up to a point. Below the ridge you are paying for weight reads you were doing anyway. Doubling the batch roughly doubles tokens per second across the batch while leaving per-user speed almost unchanged.
• Above the ridge, per-user speed degrades. Now you are contending for arithmetic, and each additional sequence takes time from the others.
• The KV cache caps the batch before the ridge does. Hundreds of concurrent long-context sequences need cache measured in hundreds of gigabytes. In practice memory capacity, not the ridge point, usually sets the batch limit — which is why paged KV allocation and cache quantisation matter so much to throughput.
• This is why a shared endpoint is efficient. One user cannot fill a batch. Many users can, and the fixed cost of reading the weights is then divided among all of them.

What actually raises the ceiling

What is not on that list: a faster clock, more arithmetic units, or a better kernel for the matrix multiply. Those help prefill, which is a real cost, but they cannot move a bound expressed entirely in bytes and bandwidth.

Confirming that you are actually against this bound rather than against something else takes one calculation and no profiler. Compute BW / (P × b) for your model and device, and compare it with the per-sequence token rate you observe. If you are within a modest factor of the ceiling, the system is working and the only remaining levers are the ones in the table above. If you are far below it, the problem is somewhere else — an empty batch, host-side overhead between steps, collective waits, or a kernel that is not using the memory system well — and optimising the model is the wrong response to any of those. The value of a derived ceiling is precisely that it distinguishes “this is physics” from “this is a bug”, which is a distinction no dashboard makes for you.