Voice Activity Detection and Endpointing

Voice activity detection asks whether the current 20 ms of audio contains speech. Endpointing asks whether the person has finished talking. They are different questions, the second is built on the first, and confusing them is why voice agents interrupt people.

Two jobs that get confused

A VAD is a per-frame binary classifier. It runs on 10, 20 or 30 millisecond frames, costs microseconds, and returns speech or not speech. It has no memory of the conversation and no opinion about whether a sentence is complete.

An endpointer is a policy on top of that stream of decisions. Its job is to decide that a turn has ended, and the simplest version — the one almost every system starts with — is a timer: when the VAD has reported non-speech continuously for min_silence milliseconds, declare the turn over. That single parameter is the most consequential number in a voice product, and it is usually left at a default.

Keep the distinction because they fail differently. A bad VAD drops the beginning of words or lets typing noise through. A bad endpointer cuts people off mid-sentence, or makes them wait after they have clearly finished. No amount of VAD accuracy fixes an endpointing policy.

How the detector works

Whichever you use, the decisions are smoothed before anything acts on them. Two parameters do that work, and they are not the same as min_silence:

• Onset debounce. Require k consecutive speech frames before declaring speech started. At 20 ms frames, k = 5 means 100 ms of sustained sound, which rejects door slams and keyboard clicks.
• Hangover. Keep declaring speech for a short period after the VAD says it stopped. Unvoiced consonants — the s in “six”, the f in “four” — have low energy and are routinely classified as silence. Without hangover you clip the ends of words, and the ASR then guesses them.
• Pre-roll. Keep a rolling buffer of the last few hundred milliseconds and prepend it when speech is detected, so the recogniser gets the attack of the first word rather than starting after it.

The endpoint sits in the critical path

The reason min_silence matters so much is that it is pure additive latency with nothing to overlap it against. Nothing downstream can start until the decision is made.

Time from the caller's last syllable to the first audio of the reply:

  gap = d_in  + e + r + m + s + d_out

    d_in    inbound network + jitter buffer, before your endpointer
            sees the silence
    e       min_silence, the endpoint wait          <-- you own this
    r       ASR finalisation after the endpoint
    m       model time to first token
    s       TTS time to first audio byte
    d_out   return network + playout buffer

Only e is a policy choice. Everything else is a system you must
either speed up or overlap.

Two levers change the sum without touching e:

  overlap r and m   start the model on the last partial transcript
                    the moment the endpoint fires, rather than
                    waiting for the final; if the final differs,
                    cancel and reissue

  overlap m and s   stream the model's tokens into the TTS as they
                    arrive, so s is paid on the first clause rather
                    than on the whole reply

With both, gap collapses towards  d_in + e + max(r, m_first) +
s_first + d_out, and e becomes the largest term you have not
addressed.

For reference on what the target should be: work on conversational turn-taking — Stivers and colleagues, PNAS 2009, across ten languages — found response offsets clustering tightly around a couple of hundred milliseconds, with remarkably little variation between languages. People notice gaps well before a second. That is the standard your system is being judged against, and the full audio-path budget adds up the rest of it.

Deriving your own threshold

The trade-off is exact and unavoidable. Within-turn pauses — hesitation, breathing, thinking, reading a number off a screen — are frequently longer than the gap you want to respond in. Any fixed threshold below the length of a pause will cut somebody off; any threshold above it makes everybody wait.

So do not adopt a number from a blog post. Measure the distribution your own callers produce, which takes one evening:

1. Run your VAD over a few hundred recorded calls, offline. Log every silence gap in milliseconds along with whether the speaker resumed afterwards or the turn genuinely ended. The ground truth for “genuinely ended” is whether the next speech in the recording came from the same speaker.
2. Build two histograms: gap lengths for within-turn pauses, and gap lengths at real turn ends. They overlap. The overlap is the irreducible part of the problem.
3. Pick the threshold as a percentile of the within-turn distribution. At the 95th percentile, you cut off one within-turn pause in twenty. At the 99th, one in a hundred, and everybody waits longer for it. State which you chose and why.
4. Segment the histogram by context. “What is your postcode?” produces a completely different pause distribution from “How can I help?”, and there is no reason to use one threshold for both.

Context-dependent thresholds are the cheapest large win available. Set a short threshold after a yes/no question, a long one after an open-ended question, and a very long one after asking for something the caller has to look up.

Judging a detector and a policy

The two jobs need two different evaluations, and using the detector’s metrics to judge the policy is how a system with an excellent VAD ends up interrupting people.

For the detector, score frames against a hand-marked reference and report the two error rates separately, because they cost different things:

• False alarm rate — non-speech frames called speech. Costs you money if you are billed per audio-minute submitted, costs you false barge-ins, and pollutes the endpointer’s view of the silence. Noise and your own echo dominate this.
• Miss rate — speech frames called silence. Costs you clipped word onsets and premature endpoints, and it is the more damaging of the two because the loss is irrecoverable: no downstream component can transcribe audio it was never sent.
• Onset and offset error, in milliseconds. More useful than either rate for this application. A detector that finds every speech region but marks its start 120 ms late is clipping the first phoneme of every utterance, and its frame-level accuracy will look fine.

For the policy, frame-level metrics are irrelevant. Score turns, against a human judgement of where each turn genuinely ended:

Plot cut-off rate against over-wait as you sweep the threshold. That curve is the actual decision, and it is specific to your callers, your questions and your audio path. Everything else in this section exists to produce it.

Semantic endpointing

The timer only sees silence. A second signal is available for free: the partial transcript. “My account number is four seven” is syntactically incomplete and a 400 ms pause after it is almost certainly mid-turn. “That is all, thanks” is complete and the same pause means the turn is over.

Implementations range from a list of trailing tokens that suppress endpointing — conjunctions, articles, prepositions, a partial digit sequence shorter than the expected length — to a small classifier over the partial text, to a dedicated turn-detection model. All of them work the same way in the end: they scale min_silence up or down per turn rather than replacing it.

effective_silence = base_silence * factor(partial_transcript)

  factor 0.6   partial ends in a completing token, or the expected
               field is fully populated ("postcode SW1A 1AA")
  factor 1.0   nothing informative
  factor 2.0   partial ends in a conjunction, preposition, article,
               or a number sequence shorter than expected
  factor 3.0   partial is empty (the caller has not started; do not
               endpoint on the silence before speech at all)

Bound the result. An unbounded factor turns one bad classification
into a caller sitting in silence.

Do not put a large language model call inside this loop. It runs on every partial and the whole point of the mechanism is to save milliseconds; a network round trip spends more than it saves. If you want a model here, it must be small enough to run locally in the audio thread.

Traps

• Your own output is speech. If the agent’s TTS reaches the microphone, the VAD will correctly detect speech and the endpointer will act on it. This is the same problem as barge-in and it needs echo cancellation, not a VAD tweak.
• Hold music and IVR tones. Music is often classified as speech by energy-based detectors and sometimes by neural ones. On a call that starts in a queue, gate on the call state, not only on the audio.
• Comfort noise. Some telephony paths transmit silence as a low-bitrate noise description rather than as real silence, and the decoder regenerates a hiss. Energy thresholds calibrated on digital silence misbehave badly on it.
• Two people in the room. A VAD detects speech, notyour caller’s speech. Background conversation restarts the timer indefinitely and the agent never gets a turn. A maximum turn length is a crude but necessary backstop.
• Frame size mismatches. Feeding a detector 512-sample chunks when it expects 480 either errors or, worse, silently misinterprets the audio. Resample and reframe explicitly rather than hoping the buffer sizes line up.