What Breaks in a Voice Assistant's Latency Budget After a Migration

The complaint after a voice migration is never a number. It is “it feels laggy now”, or barge-in stopped working, or testers report the assistant talking over them. The margin that used to absorb the model’s share of the turn stopped absorbing it, and the dashboard says nothing changed.

The symptom, and the metric that misses it

A spoken turn is a chain of budgets: the endpointing delay before your voice activity detector decides the user stopped, the tail of streaming transcription, the model’s time to first useful token, the synthesiser’s time to first audio, and playback buffering. Only one link moved. But the whole chain is judged by a listener against a single perceived gap, so a few hundred milliseconds added in the middle reads as a broken product.

The reason the dashboard is quiet is usually the definition of the metric. Most time-to-first-token instrumentation records the moment the HTTP response starts producing bytes, or the moment the first server-sent event is parsed. Neither is when speech can start. Both providers open a stream with an event that carries metadata and no content — a message-start event carrying the message id and initial usage on the Messages API, an opening chunk carrying the assistant role and an empty delta on an OpenAI-shaped one. A probe that stops at the first byte measures the connection, not the model, and will show no regression at all while users hear one.

Reasoning tokens arrive before any speech

The largest single contributor to a post-migration voice regression is a model that thinks before it speaks. Reasoning or thinking tokens are generated first and are not part of the visible answer, so the first text delta can arrive well after the stream opened.

This is not a subtle inference. Anthropic’s own migration documentation states that on its current Opus model thinking is on by default when the parameter is omitted — a change from the previous generation, where omitting it meant no thinking — and that with the default display setting the thinking blocks stream with empty text, which it describes as looking like a long pause before output to a streaming interface. An adapter that stopped sending the old provider’s reasoning-effort parameter and did not set anything in its place has therefore turned thinking on without anyone choosing to.

The same applies in the other direction on OpenAI-shaped APIs, where reasoning-tier models spend reasoning tokens that are billed as output and reported under usage.completion_tokens_details.reasoning_tokens but never appear in the text. If that number is non-zero on a voice turn, you are paying for latency the listener experiences as silence.

For a voice turn the correct setting is almost always the lowest reasoning depth the task tolerates, or none. Reserve deliberation for the turns that need it — a tool-planning turn the user is not waiting on in real time can afford it; the turn that produces the spoken reply cannot.

Prefill, tokenizers and a cold cache

Three smaller mechanisms compound with the first, and all three are consequences of the migration rather than of your code.

• Time to first token scales with input length. The model must process the whole prompt before emitting anything. A voice assistant with a long system prompt and a growing conversation history pays that on every turn, and the effect grows through the conversation.
• The same string is a different number of tokens. Tokenizers differ between model families, and vendors change them between generations. Anthropic’s migration notes for one recent generation describe the same content tokenising to materially more tokens than on the previous one. Your prompt did not grow; its prefill did.
• The cache is cold, and may stay cold. If your old deployment relied on a cached system prefix to keep prefill short, the first turn of each session on the new provider pays full price — and if the prefix falls below the new provider’s minimum cacheable length it never caches at all, silently. The mechanics are in re-deriving caching projections.

Regional routing belongs on the list too: if the new provider’s endpoint you are calling is further from your inference tier than the old one, every turn carries the extra round trip, and no prompt change will recover it.

The four timestamps to instrument

The fix starts with measuring the right interval. Record four timestamps per turn and emit all four, not a single derived duration:

1. Request sent. The moment the last byte of the request leaves your process, after prompt assembly.
2. First stream event received. The metadata event. Useful only as a measure of connection and queueing time.
3. First text delta. The first event that carries actual visible content. The interval from the second to the third timestamp is the reasoning and prefill cost, and it is the number that regressed.
4. First synthesisable unit. The point at which enough text has accumulated for the synthesiser to start — usually a clause or a sentence boundary rather than a single token. This is the real budget item, because it is when audio can begin.

Compare percentiles, not means, and compare the same percentile before and after. A mean hides the case that generates the complaints: a bimodal distribution where most turns are fine and one in twenty thinks. Take the 95th percentile of the fourth timestamp minus the first, and re-derive the budget for the rest of the chain from what is left of your target turn gap.

Levers, in order of effect

Apply them in this order, because the first two usually make the rest unnecessary. Turn reasoning off or to its lowest setting for the spoken turn, and verify by checking that the reasoning-token count in the usage object is zero rather than by listening. Then shorten what the model must read before it can speak: trim the system prompt, truncate the history aggressively for the speaking turn, and make sure the stable part is above the provider’s minimum cacheable length so it is actually cached.

After that, start speaking sooner rather than making the model faster. Synthesising at the first clause boundary instead of the first full sentence recovers a large fraction of the perceived gap and costs nothing but a small change to your buffering rule. Constrain the answer length so the model does not open with a preamble the listener has to sit through. Only when those are exhausted should you add a filler sound or an acknowledgement token to cover the gap, because that hides the symptom and makes the next regression invisible.