Real-Time Voice Agents: The Full Latency Budget
7 min read · updated August 3, 2026
A voice agent feels either alive or broken, and the boundary is a few hundred milliseconds wide. The terms in that budget are listed below; the values are yours to measure. What can be said without measuring anything is which terms are large, which are irreducible, and which are under your control.
What number are we aiming at
The target is not arbitrary and does not come from product intuition. Conversation analysis has measured the gap between turns in ordinary human speech for decades; Stivers et al. reported in PNAS (2009) that across ten languages from unrelated families the modal between-turn gap clusters near 200 ms, with the distributions overlapping strikingly despite enormous linguistic differences. Humans are also strikingly sensitive to deviation from it — a delay of a second reads as hesitation, reluctance, or a bad line.
So the design target for a natural-feeling agent is a response beginning within roughly half a second of the user finishing, and the practical goal is usually to get under a second. Above about two seconds users start talking again over the top of the agent, which creates a second problem on top of the first.
The terms
| Stage | Description |
|---|---|
| 1 · capture | Microphone buffer plus codec frame size. Opus uses 20 ms frames by default; some stacks buffer several. Small, fixed, not worth attacking. |
| 2 · uplink | Half your RTT. Fixed by geography and the user's connection. Measurable, not reducible except by moving the endpoint closer. |
| 3 · endpointing | The silence you wait for before deciding the user stopped. Typically the largest term in the whole budget. Entirely yours. |
| 4 · recognition finalisation | Pipeline designs only: the delay between the last partial hypothesis and the punctuated final one. |
| 5 · model prefill | Time to first token, which grows with the conversation history you resend every turn. |
| 6 · first sentence | Enough output tokens to have a clause worth speaking. Set by the model's token rate and by where you cut. |
| 7 · synthesis | Time to first audio from the TTS. Zero for native-audio designs, which emit speech tokens directly. |
| 8 · downlink + jitter buffer | Half RTT plus client buffering. A conservative 200 ms jitter buffer silently doubles a good result. |
Stages 3 through 7 are sequential and they add. This is the reason architectural choices dominate: removing a stage removes its whole term, while making a stage faster shaves a fraction of one.
Endpointing is the big one
Endpointing is the decision that the user has stopped talking. The naive implementation waits for a fixed silence — 500 to 800 ms is a common default — and that silence is pure latency added to every single turn, larger than anything else in the table.
You cannot simply set it to 100 ms, because people pause mid sentence. “My account number is four two … seven …” will be cut off, the agent will interrupt, and the user will have to start again — which costs several seconds and feels far worse than waiting. The tension is real and it is why this term is engineering rather than configuration.
What actually works is making the threshold adaptive on evidence you already have:
- Syntactic completeness. A partial transcript ending in “and”, “because” or a bare digit is almost certainly unfinished; one ending in a complete clause probably is not. Wait longer for the former.
- Falling pitch. A dropping pitch contour is a strong end-of-turn cue in most languages and is available from the audio without any model call.
- Context. After “yes or no?” you can be aggressive. After “tell me what happened” you should not be.
- Speculate. Start the model call on the current partial transcript and cancel it if more speech arrives. You spend some wasted tokens to buy back most of stage 5, and the trade is usually favourable.
Attacking the budget in order
Fill in your own numbers for stages 1–8, then work top-down by size. The order in which the wins usually appear:
- Tune endpointing. Largest single term, no vendor required, no cost.
- Stream and cut at the first sentence. Removes the model’s full generation time from the critical path.
- Collapse stages. A native audio model removes stages 4 and 7 outright. This is the biggest architectural lever available and the main reason those models exist.
- Shorten the prompt, cache the prefix. Stage 5 grows with conversation length, so a voice call gets slower as it goes on unless the history is summarised or the prefix is cached.
- Move the endpoint. Only after the above; it is the expensive fix for the smallest fixed terms.
Covering what you cannot remove
Some latency will survive, and perceived latency is the thing users actually judge. Two techniques buy a great deal of it. A short filler — “let me check that” — emitted the instant endpointing fires, from cached audio, starts the response at close to zero while the real answer is still being generated. And where the agent calls a slow tool, saying so out loud converts dead air into a status update. Neither makes the system faster. Both make it feel like a conversation rather than a fault.
Barge-in belongs in the same category and is not optional. If a user starts talking while the agent is speaking, the agent must stop within a couple of hundred milliseconds — which means the client detects speech locally rather than waiting for a server round trip, and whatever audio is already sitting in the playback buffer is discarded rather than drained. It also means the conversation state has to record what the agent actually said before it was cut off, not what it intended to say, or the next turn is answered against a history the user never heard. That last detail is the one most implementations get wrong, and it produces a maddening failure in which the agent behaves as though it had already told you something.