Prompt Tokens, Completion Tokens and Why They’re Priced Apart

Every provider charges more for output than input, usually by a factor of three to five, and occasionally by far more. It is not a margin decision. The two halves of a request run on the hardware in fundamentally different regimes, and the arithmetic is worth doing once.

One request, two different machines

Serving a request has two phases. Prefill reads your entire prompt and builds the key/value cache for it. Every prompt token is known in advance, so they all go through the network in one batched pass. Decode then produces the answer one token at a time, each step depending on the token before it, with no way to parallelise across the sequence.

Same weights, same model, completely different bottleneck. That is the whole explanation, and the rest of this page is the numbers behind it.

Prefill is compute-bound

A forward pass costs roughly 2 × params floating-point operations per token — one multiply and one add per parameter. For a 70B-parameter dense model and a 2,000-token prompt:

FLOPs = 2 x 70e9 x 2000 = 2.8e14 = 280 TFLOP

NVIDIA publishes ~990 TFLOP/s dense BF16 for an H100 SXM.
At a realistic 40% utilisation:  ~400 TFLOP/s

  280 / 400 = 0.70 s for the whole 2,000-token prompt
            = ~2,850 prompt tokens per second

The important property is that this scales with the batch. The hardware is doing dense matrix multiplications, which is what it is built for, so utilisation is high and the cost per prompt token is low. Doubling the prompt roughly doubles the work, and that work is cheap per unit.

Decode is bandwidth-bound

Now generate one token. The arithmetic is trivially small — one token through the network — but to do it the device must read every weight it needs out of memory. For a single sequence, the entire model gets streamed from HBM once per token:

70B params at BF16 = 140 GB of weights
H100 SXM HBM3 bandwidth (published): ~3.35 TB/s

Split across 4 GPUs with tensor parallelism, each reads 35 GB:
  35 GB / 3.35 TB/s = 10.4 ms per decode step
                    = ~96 tokens/s per sequence, as a ceiling

Arithmetic done in that 10.4 ms: 2 x 70e9 = 140 GFLOP
Device capability in 10.4 ms:    ~10,300 GFLOP

Utilisation of the expensive part: about 1.4%.

That last line is the crux. During decode the accelerator is almost entirely idle in compute terms and completely saturated in memory terms. Batching is the only fix: run 64 sequences together and the same single read of the weights serves all 64, so throughput per device rises nearly linearly while per-sequence speed stays roughly flat. This is why providers batch aggressively, and why the KV cache — which does not amortise, since each sequence has its own — becomes the binding constraint on how large a batch can get.

The KV cache deserves a moment of its own, because it is the term that couples the two phases. Every token in the sequence, prompt and generated alike, contributes key and value vectors that must be kept for the rest of the request. Its size grows with context length, with the number of layers and with the number of key/value heads — which is exactly why grouped-query and multi-query attention exist, since sharing key/value heads across query heads shrinks the cache by the sharing factor and buys back batch slots. A long prompt therefore slows decoding slightly as well as prefill, not because the weights take longer to read but because there is more cache to read alongside them.

Why the ratio lands where it does

Put the two together. Per token of work, prefill runs at high utilisation on hardware that is good at it; decode runs at low utilisation on hardware that is waiting for memory, and it holds a KV cache slot for the whole generation. The cost of an output token, in device-seconds, is far above that of an input token, and the published three-to-five-times price ratio is if anything a conservative reflection of the underlying gap, softened by batching.

Two corollaries worth carrying around. First, this is why long context lengthens time-to-first-token but not the tokens-per-second you see afterwards. Second, it is why prompt caching produces such a large discount: cached prefill is not merely cheaper compute, it is compute that does not happen at all — see cached tokens.

It also explains why the industry’s main latency research programme is about breaking the one-token-per-pass constraint rather than about faster hardware. Speculative decoding is the clearest example: a small draft model proposes several tokens, the large model verifies all of them in a single pass, and any prefix the large model agrees with is accepted. Verification is a prefill-shaped operation — compute-bound, parallel, cheap per token — so the technique converts expensive decode steps into cheap prefill work without changing the output distribution. That it works at all is the strongest available evidence that decode was never limited by arithmetic.

What to do with this

• Optimise output first. A 30% cut in output length is worth several times a 30% cut in input length, on both cost and latency. Structured output instead of prose is the usual mechanism.
• Do not fear a long, stable prompt. Input is the cheap side and the cacheable side. Many-shot examples that reliably shorten the answer can pay for themselves.
• Price your feature on the output distribution, not the mean. Output length is usually long-tailed; the p95 answer drives both your worst latency and a surprising share of spend.
• Watch for models that break the ratio. Reasoning models bill hidden thinking as output, so their effective output volume is much larger than the visible answer suggests.

One caveat on all of the above: it describes dense models on accelerators, which is the common case and not the only one. Sparse mixture-of-experts models read only the active experts per token, so the bandwidth term falls while the memory footprint does not. Quantisation moves the same term — halving the bytes per weight roughly halves the read. The structure of the argument survives both; only the constants move, which is why the input-to-output price ratio is stable across families that differ enormously in every other respect.