Prefill vs Decode: The Two Halves of Inference

Every provider charges more for output tokens than input tokens, usually three to five times more. That is not a pricing preference. It falls out of the fact that the two halves of inference hit different hardware limits, and you can derive the ratio yourself from a datasheet.

Two phases, one model

Prefill processes the prompt. Every token in it is already known, so all of them go through the model together, as one large matrix multiplication per layer. Output: the KV cache for the whole prompt, plus the first token’s logits.

Decode produces the answer. Each step handles exactly one new token, attends to everything cached so far, and cannot start until the previous step finished. Output: one token, and one more column in the KV cache.

Same weights, same layers, same arithmetic per token. The difference is how many tokens are in flight at once, and that difference is everything.

Arithmetic intensity, and why it decides

Arithmetic intensity is the ratio of floating-point operations performed to bytes moved from memory. It decides which of the two hardware limits you hit: a low ratio means the chip is idle waiting for weights to arrive (memory-bound), a high one means memory is idle waiting for the maths to finish (compute-bound).

Take a dense model with P parameters stored in 2-byte precision. A forward pass over N tokens costs roughly 2·N·P FLOPs — the factor of two is one multiply and one add per parameter per token — and it must read 2·P bytes of weights, once, regardless of N. So:

intensity(N) = FLOPs / bytes
             = (2 * N * P) / (2 * P)
             = N            FLOPs per byte

prefill, N = 4000 tokens   ->  ~4000 FLOP/byte
decode,  N = 1 token       ->  ~1 FLOP/byte

That is the whole argument, and it is exact to within the terms we dropped (attention itself, which adds work that grows with sequence length, and the KV cache reads, which matter at long context). Prefill reads the weights once and does thousands of operations with each byte. Decode reads the same weights and does one.

Putting a real GPU in the equation

A chip has a break-even intensity: its peak FLOP/s divided by its memory bandwidth. Below that ratio you are memory-bound, above it you are compute-bound. NVIDIA’s published H100 SXM specification quotes roughly 990 TFLOP/s of dense BF16 throughput and 3.35 TB/s of HBM3 bandwidth. Divide:

break-even = 990e12 FLOP/s / 3.35e12 byte/s
           ~= 295 FLOP per byte      (NVIDIA H100 SXM datasheet figures)

Compare with the two intensities above. Prefill at a few thousand tokens sits far above 295 and is compute-bound: it uses the expensive part of the chip properly. Decode at one token per step sits at roughly 1, which is more than two orders of magnitude below the break-even point — during decode a card of that class is spending essentially all of its time streaming weights out of HBM and almost none of it computing.

That is why single-stream decode speed is, to a good approximation, bandwidth / model_bytes tokens per second, and why it barely improves when you move to a chip with more FLOPs but similar memory bandwidth.

Batching only rescues one of them

The fix for a memory-bound kernel is to do more work per byte read. Decode has an obvious source of extra work: other users. If B sequences decode in the same step, the weights are read once and used B times, so intensity becomes roughly B instead of 1. At B around 256 you are back near the break-even ratio and the hardware is being used well.

This is why serving economics are what they are. Decode is cheap per token only when it is shared, and sharing means your request sits in a batch with strangers, which is precisely the source of the latency variance in continuous batching. Prefill needs no such rescue — it is already compute-bound with a batch of one — which is also why long prompts from different users compete with each other for the same scarce resource.

There is a further consequence that is reshaping how large deployments are built. If the two phases want opposite things from hardware — prefill wants FLOPs and is happy alone, decode wants bandwidth and needs company — then running them on the same device forces a compromise on both. Disaggregated serving splits them onto separate pools: prefill workers compute the prompt’s KV cache, ship it over a fast interconnect, and decode workers do nothing but generate at high batch. Each pool can then be sized, scheduled and even specified in hardware for the bottleneck it actually has.

You cannot see any of this from the API, but it explains an otherwise odd observation: on some endpoints, time to first token and tokens per second appear to move independently under load rather than degrading together. That is what two separately-scheduled pools look like from the outside.

What follows for your bill

• The price ratio is a cost ratio. Output tokens occupy a memory-bound pipeline for a whole step each; input tokens ride along in a compute-bound pass thousands at a time. Charging the same for both would mean subsidising output out of input.
• A long prompt is cheaper than it feels. Ten thousand input tokens is one efficient pass. Five hundred output tokens is five hundred sequential ones. Trading prompt length for output length — few-shot examples that make the model answer tersely — is usually a win on both time and money.
• Prompt caching attacks prefill only. A cache hit skips recomputing the prefix’s KV entries. It cannot make decode faster, so it moves time to first token and not tokens per second.
• Quantisation attacks decode. Halving the bytes per weight roughly halves the bytes decode must stream, which is the binding constraint. That is why 8-bit and 4-bit weights show up in serving long before they show up in training.