Time to First Token vs Tokens per Second

A single “latency” figure for a streaming model is an average of two things that move independently and are fixed by unrelated means. Which of the two you should be optimising is not a matter of taste — it is determined by whether a human is reading the output as it arrives.

Two numbers, defined precisely

Note what is missing from each. Halving your system prompt does nothing to TPS. Asking for a shorter answer does nothing to TTFT. If you are reporting one number you cannot tell which lever to pull, and the usual outcome is pulling the wrong one and concluding the optimisation did not work.

The reader is slower than you think

For anything a person watches arrive, there is a threshold above which extra generation speed is invisible, and it is low. Adult silent reading of ordinary prose runs somewhere around 200–300 words per minute in the psycholinguistics literature; take 250 as a working assumption. English tokenises at roughly 0.75 words per token for common vocabulary. So:

reader demand = 250 words/min / 0.75 words/token / 60 s
              ~= 5.6 tokens/second        (assumption: 250 wpm, 0.75 w/tok)

Roughly six tokens per second keeps ahead of a reader. Hosted models typically stream at many multiples of that. Which means, for a chat interface, additional TPS beyond a fairly low bar buys nothing a user can perceive, while every millisecond of TTFT is dead air they are staring at.

The caveat is honest: readers skim, they scroll back, and text arriving in visible bursts feels worse than the same tokens arriving evenly. But the order of magnitude holds. If your users are reading, TTFT is the number.

Evenness is worth treating as its own property rather than folding into the rate. A stream delivering 60 tokens per second in bursts of twenty every third of a second has the same average as one delivering them smoothly, and reads considerably worse — the eye notices the stalls, not the throughput. Two things produce this: batch-composition effects on the server, and buffering in your own transport that accumulates chunks before flushing. Only the second is yours, and it is worth checking, because a small write buffer somewhere in the path is a common and entirely fixable cause of a stream that feels laggy at a perfectly healthy rate.

When TTFT stops mattering entirely

Now remove the human. An agent loop that makes six model calls in sequence, each producing 400 tokens, and shows the user only the final answer, has a completely different arithmetic:

6 steps x (TTFT 0.4 s + 400 tokens / TPS)

TPS = 30   ->  6 * (0.4 + 13.3)  =  82 s        TTFT is 3% of it
TPS = 90   ->  6 * (0.4 + 4.4)   =  29 s        TTFT is 8% of it

Generation speed dominates, TTFT is rounding error, and the numbers above are illustrative arithmetic from assumed rates rather than measurements of any provider. Swap in your own step count and token counts; the structure of the answer does not change. Nobody is watching the intermediate tokens, so streaming them faster is the only thing that shortens the wait.

The same applies to any non-streaming consumer: a classification job, a batch enrichment, a tool that parses JSON out of the response. If the first token is never displayed, its arrival time has no user-visible meaning at all.

A third number, for reasoning models

Models that spend tokens thinking before answering break the definitions above, and the break is not cosmetic. The first token to arrive may be the first reasoning token, which the interface either hides or shows in a separate panel. Time to first token is then measuring the start of something the user is not reading, and the number they actually feel is the one nobody records:

time to first ANSWER token
  = ttft + reasoning_tokens / rate

reasoning budget 1500 tokens, rate 60 tok/s, ttft 0.5 s
  ->  0.5 + 25  =  25.5 s of a blank answer pane

Twenty-five seconds is not a latency problem you can fix by tuning prefill. It is a product problem, and the available answers are product answers: show the reasoning stream so the wait is legible, show intermediate structure, or spend fewer thinking tokens. Where the API exposes a reasoning-effort or thinking-budget control, that control is your largest latency dial by a wide margin — larger than model choice and larger than prompt length.

The measurement consequence: for a reasoning model, record three timestamps rather than two — first token, first non-reasoning token, and end. Report the middle one as the perceived latency, and treat a dashboard that has only the first as measuring something nobody experiences.

A decision procedure

• Is a human reading tokens as they arrive? If yes, optimise TTFT: shorten and cache the prompt prefix, pick a region close to the user, reuse connections, and stream from the first byte rather than buffering in a proxy. Accept any TPS above roughly ten.
• Is the output consumed whole? Then only total time exists. Optimise output length first — it is the term you control most directly — then TPS, then TTFT last.
• Is it a multi-step agent? Multiply everything by the step count, then reduce the step count. One fewer round trip beats any per-call tuning.
• Is there no deadline at all? Then neither number is your metric; cost per task is, and an asynchronous batch endpoint usually wins outright.

How these get misreported

Three failure modes account for most confusing latency numbers. A buffering proxy makes TTFT equal total time, so a perfectly healthy model looks catastrophically slow — check that your CDN, ingress and any framework response wrapper all pass chunks through unbuffered. Averaging TPS across a request that stalled mid-stream hides the stall entirely, because the mean rate recovers even though the user saw a three-second freeze; report inter-token maxima as well as the mean. And quoting a mean rather than a percentile lets one queue-delayed request define a figure nobody experienced.

Report four numbers per request and every one of those disappears: TTFT, total duration, output token count, and the largest gap between consecutive tokens.