Latency Budgets for Reasoning Models

For a normal completion, time to first token tells you when the user starts reading and the token rate tells you the rest. Insert a thousand-to-fifteen-thousand-token thought in front of the answer and the first number becomes almost meaningless while the second becomes almost everything.

The formula, with the new term

A conventional completion is roughly total = prefill + output_tokens / rate, and the user sees the first token after prefill. A reasoning completion inserts a term before the visible output:

total          = prefill + (thinking_tokens + answer_tokens) / rate
visible_start  = prefill + thinking_tokens / rate     // if trace hidden
               = prefill                              // if trace streamed

Put numbers on it. Take a generation rate of 60 tokens per second, which is an unremarkable figure for a mid-sized model on current serving hardware, a prefill of 400 ms, a 5,000-token trace and a 250-token answer. Total is 0.4 + 5250/60, or roughly 88 seconds. If the trace is hidden, the user stares at nothing for 84 of those seconds and then reads for four.

That is the whole of it, and every operational consequence below comes out of that one arithmetic.

Note which term you can influence. The rate is the provider’s; the prefill is roughly proportional to your prompt and is rarely the problem. The thinking count is the only large term you control, and you control it with the effort or budget setting rather than with anything about the prompt. That makes latency tuning on a reasoning model a much shorter conversation than on a conventional one — there is essentially one dial, and it trades directly against accuracy.

Why TTFT stopped being useful

Time to first token was a good metric because it correlated with two things at once: how fast the service was, and how soon the user was no longer waiting. On a reasoning model those decouple completely.

If the provider streams thinking, TTFT stays small and tells you nothing about when the answer arrives. If the provider hides thinking, TTFT balloons into a number that is mostly a measure of how hard the model found the question — so a model that scores worse on your TTFT dashboard may simply have been given a harder prompt mix. Either way, it has stopped being a service-health signal.

Replace it with two metrics that survive: time to first visible answer token, which is what a human experiences, and total request time, which is what your infrastructure has to survive. Track both at p95 and p99. A mean is useless here for the reason means are always useless on a long-tailed distribution: the tail is not noise, it is the hard questions, and the hard questions are the ones you deployed a reasoning model for.

The variance is the actual problem

A conventional model’s response length is bounded by your instructions and by max_tokens, and its distribution is fairly tight. Thinking length is bounded by the budget and by the model’s own judgement, and its distribution is not tight at all: the same prompt template can produce a 300-token trace on an easy instance and a 12,000-token trace on a hard one.

So a capacity plan built on averages fails in a specific way. You size your worker pool for a 20-second mean, a batch of genuinely difficult inputs arrives together — which they do, because difficulty is correlated with whatever caused the batch — and every worker is occupied for two minutes at once. The queue behind them is now longer than any timeout you set. Size on p95 duration and cap concurrency per worker at one; a reasoning request holds a connection open far longer than the CPU work it represents, so the usual justification for overcommitting does not apply.

Retries deserve a specific warning. The standard resilience pattern — retry once on a timeout or a 5xx — was designed around calls that fail in a second or two. Applied to a ninety-second request it produces a three-minute worst case and bills you for both attempts, including all the thinking tokens of the one you discarded. If you retry reasoning calls at all, retry at a lower effort level, and never retry on a deadline you set yourself: your own timeout expiring is not evidence that the provider failed, and re-issuing the identical expensive request is the least likely thing to fix it.

Things that time out before you do

Long requests collide with defaults that were chosen when a slow API call meant three seconds. The specific numbers to check, all of them documented by their vendors:

The single most effective mitigation is to stream, even if you buffer the tokens and display nothing until the end. Streaming turns one long silence into a continuous trickle of bytes, and almost every timeout in the list above is an idle timeout rather than a total one. The UX side of the same decision is covered in streaming partial thoughts.

Setting a budget that holds

• Work backwards from the human. Decide what the user will tolerate, subtract your own overhead, convert what remains into tokens at your observed rate, and set the thinking budget to that number. Budgets derived from token counts you liked the look of do not survive contact with a p99.
• Make the timeout a real deadline, not a guess. If the answer is worthless after 30 seconds, cancel at 30 seconds and fall back. Waiting 90 seconds for something nobody will read is the worst available outcome: you pay for every one of those tokens.
• Move it off the request path. If the task genuinely needs a minute of thinking, it is a job. Return an identifier, notify on completion, and stop trying to make an HTTP request survive something it was not built for.