Autoregressive Generation: Why LLMs Write One Token at a Time

A model produces one token per forward pass. Everything about the latency of a language model application follows from that one fact.

The loop

To generate an answer, the runtime does this: feed the prompt, get a distribution, sample a token, append it to the sequence, feed the whole thing again. Repeat until the model emits a stop token or you hit max_tokens. A 300-token answer is 300 iterations of that loop, and each one depends on the result of the last — which is what autoregressive means, and why the work cannot be spread across more hardware to go faster.

Why input is fast and output is slow

Reading the prompt is one pass over all of it at once. Every token in the prompt is known up front, so the whole sequence goes through the model in parallel — this is the prefill, and it is bounded by how much arithmetic the hardware can do.

Generating is the opposite. Each step produces one token and must load the model’s weights to do it, so the bottleneck is memory bandwidth rather than arithmetic. A 10,000-token prompt is often cheaper in wall-clock time than a 500-token answer, and this is why providers price input and output separately — usually with output several times dearer. That price difference is not a margin decision. It reflects a real difference in what the two operations cost to serve.

Time to first token is a different number

Two latencies matter and they behave differently. Time to first token is dominated by prefill, so it grows with prompt length. The rate after that — tokens per second — is roughly flat for a given model on given hardware, so total time is approximately ttft + (output_tokens / rate).

Averaging them into one “latency” number hides which one you have a problem with. A long system prompt hurts the first; a verbose answer hurts the second; and the fixes are unrelated.

What you can actually do about it

• Stream. It does not make generation faster, but the reader starts at time-to-first-token instead of at the end. For anything a human watches, this is the largest perceived win available.
• Ask for less. Output tokens cost time and money in direct proportion. “Answer in one sentence” is a latency optimisation as much as a style one.
• Cache the prompt. Prefill is the part that can be reused. Where a provider supports prompt caching, a long shared prefix is charged at a reduced rate and skips most of its own recomputation.
• Pick for the shape of the work. A small fast model that needs two passes can beat a large one that needs a single slow pass — and can lose badly if it needs five. This is measurable for your workload and guessable for nobody else’s.