Barge-In: Letting a Caller Interrupt

Barge-in means the caller can talk over the agent and the agent stops. It sounds like one if statement. It is four problems, and the one nobody warns about is that stopping the synthesiser does not stop the sound.

Why it is harder than it looks

The naive implementation is: while speaking, run the VAD on the inbound stream; if it reports speech, stop the TTS. Deployed on a phone call, that produces an agent that interrupts itself constantly, because everything below is true at once.

• The agent’s own voice comes out of the caller’s earpiece, into the caller’s microphone, and back down the line. The VAD detects it, correctly, as speech.
• The mechanism that removes that echo can also remove the caller’s genuine interruption, because it was designed to prioritise one direction at a time.
• When you do decide to stop, up to a second of already-synthesised audio is sitting in buffers between your process and the caller’s ear, and it will play regardless.
• A cough, a door, or the word “mm” is speech. Cancelling a twenty-second explanation because somebody cleared their throat is a worse experience than not supporting barge-in at all.

Your own voice arriving back

There are two distinct echo sources and they need different answers.

The important structural point: acoustic echo cancellation needs the far-end reference signal, so it can only run where both signals exist. For a WebRTC caller that is their browser, which does it automatically and well. For a PSTN caller it is their handset and the network, and you have no control over either. So a server-side agent cannot cancel acoustic echo — it can only detect it and refuse to act on it.

The server-side control that does work is straightforward and is what most robust systems actually use: you know exactly what you are sending. While the agent is speaking, you have the outbound audio, the time you sent it, and the estimated round trip. Inbound speech that correlates with your own recent output — or simply arrives during a window when you are speaking and disappears within a hundred milliseconds of you stopping — is echo, not a caller. Suppressing on that basis costs nothing and removes the most common false barge-in.

The half-duplex trap

Echo cancellers do not only subtract. When the residual is large — a loud speakerphone, a badly modelled path, a nonlinear amplifier — the canceller falls back to suppression, attenuating the near-end signal while the far end is active. That is the classic half-duplex behaviour people recognise from old speakerphones: only one person can be heard at a time.

For barge-in this is fatal, and it is fatal silently. The caller interrupts, their audio is attenuated below your VAD threshold by something in the path you do not control, the agent keeps talking, and the caller repeats themselves louder. Nothing in your logs shows a problem: the VAD simply did not fire.

• Lower the barge-in threshold specifically while speaking. A separate, more sensitive detection threshold during the SPEAKING state compensates for the attenuation, and the extra false positives are absorbed by the debounce policy below.
• Duck rather than stop, first. Dropping your output level by 6–12 dB the instant you suspect speech removes the suppression pressure and lets the caller’s audio through cleanly, so the confirmation decision is made on a better signal. Ducking is reversible; stopping is not.
• Prefer wideband and headsets where you have the choice. A WebRTC caller on a headset has essentially no acoustic echo, so barge-in that works badly on a PSTN speakerphone may work perfectly there. Do not conclude from a browser test that the phone path is fine.
• Keep utterances short. The most reliable barge-in fix is to give the caller frequent natural gaps in which they do not need to interrupt. A twenty-second monologue is the problem; barge-in is a mitigation for it.

The audio already in flight

This is the part that is missing from nearly every tutorial, and it is the reason a correct-looking implementation still talks over people for most of a second.

When you decide to stop, your synthesised audio exists in
several places at once:

  1. the TTS response stream you have not finished reading
  2. your application's outbound buffer
  3. the media server's send queue
  4. packets in flight on the network
  5. the caller's jitter buffer
  6. the caller's device playout buffer

Calling stop() on the synthesiser addresses (1). It does nothing
about (2) through (6), and their combined depth is commonly
several hundred milliseconds -- easily more if any buffer is
generously sized for reliability, which most are by default.

So the sequence has to be, in this order:

  a. stop reading from the TTS and cancel the request, so you
     stop paying for audio nobody will hear
  b. clear your own outbound buffer
  c. tell the media server to flush its queue for this call --
     if its API has no flush, this is a blocking limitation and
     you need to know that before you choose it
  d. optionally send a short burst of silence or comfort noise,
     so the caller hears a clean cut rather than a truncated
     syllable
  e. remember WHERE you stopped in the text, so the agent can
     say "sorry, go on" rather than resuming from the beginning

Steps (5) and (6) belong to the caller's device and cannot be
flushed at all. That residual is your floor: even a perfect
implementation keeps talking for the depth of the far-end
buffers. Measure it once with a real handset and stop trying to
beat it.

Step (c) is worth checking before you pick a telephony platform. “Can I flush queued outbound audio mid-utterance?” is a question with a yes-or-no answer, and a platform that answers no cannot do responsive barge-in no matter what you write.

The state machine

Barge-in only stays correct if turn-taking is an explicit state machine. Scattering flags across event handlers is how you get an agent that ends up speaking and listening at the same time, or neither.

// turn-machine.ts -- no dependencies.
//
// States:
//   IDLE      nothing happening, not listening for a turn
//   LISTENING caller has the floor
//   THINKING  endpoint fired; ASR/model/TTS in flight; not speaking
//   SPEAKING  audio going out
//   DUCKED    speaking, but suspected barge-in: output attenuated
//             while we confirm

export type State = "IDLE" | "LISTENING" | "THINKING" | "SPEAKING" | "DUCKED";

export interface Audio {
  setGain(db: number): void;
  /** Cancel TTS, clear local buffer, flush the media server queue. */
  flushOutbound(): Promise<void>;
  /** Characters of the current utterance already sent for synthesis. */
  spokenOffset(): number;
}

const FRAME_MS = 20;
const CONFIRM_MS = 240;   // sustained speech required to cancel
const DUCK_MS = 60;       // speech required to duck (cheap, reversible)
const DUCK_DB = -9;

export class TurnMachine {
  state: State = "IDLE";
  private voicedMs = 0;
  private unvoicedMs = 0;
  private utteranceText = "";

  constructor(
    private readonly audio: Audio,
    private readonly on: {
      interrupted(spokenPrefix: string): void;
      turnEnded(): void;
    },
    /** min_silence, from your own pause histogram. */
    private readonly endpointMs = 600,
  ) {}

  beginSpeaking(text: string) {
    this.utteranceText = text;
    this.voicedMs = 0;
    this.unvoicedMs = 0;
    this.state = "SPEAKING";
  }

  finishedSpeaking() {
    if (this.state === "SPEAKING" || this.state === "DUCKED") {
      this.audio.setGain(0);
      this.voicedMs = 0;
      this.unvoicedMs = 0;
      this.state = "LISTENING";
    }
  }

  /** Called once per inbound audio frame. voiced comes from the VAD. */
  async onFrame(voiced: boolean) {
    if (voiced) {
      this.voicedMs += FRAME_MS;
      this.unvoicedMs = 0;
    } else {
      this.unvoicedMs += FRAME_MS;
      // A gap resets the barge-in evidence, so a cough cannot
      // accumulate towards CONFIRM_MS across two seconds.
      if (this.unvoicedMs >= 3 * FRAME_MS) this.voicedMs = 0;
    }

    switch (this.state) {
      case "SPEAKING":
        if (this.voicedMs >= DUCK_MS) {
          this.audio.setGain(DUCK_DB);
          this.state = "DUCKED";
        }
        return;

      case "DUCKED":
        if (this.voicedMs >= CONFIRM_MS) {
          const prefix = this.utteranceText.slice(0, this.audio.spokenOffset());
          await this.audio.flushOutbound();
          this.audio.setGain(0);
          this.state = "LISTENING";
          this.unvoicedMs = 0;
          this.on.interrupted(prefix);
        } else if (this.voicedMs === 0) {
          // Evidence evaporated: it was echo or a noise. Un-duck.
          this.audio.setGain(0);
          this.state = "SPEAKING";
        }
        return;

      case "LISTENING":
        if (this.voicedMs > 0 && this.unvoicedMs >= this.endpointMs) {
          this.state = "THINKING";
          this.on.turnEnded();
        }
        return;

      case "THINKING":
      case "IDLE":
        return;
    }
  }
}

Three properties of that machine are the ones doing the work. DUCKED exists so the expensive, irreversible decision is made on a cleaner signal than the one that triggered it. The reset of voicedMs after three unvoiced frames means evidence must be contiguous, so a cough cannot accumulate. And interruptedreceives the prefix that was actually spoken, so the next turn can reference what the caller heard rather than what you generated — the difference between an agent that repeats itself and one that does not.

Deciding what counts as an interruption

Duration is the cheap signal. Content is the better one, and you have it: while ducked you are already streaming the caller’s audio to the recogniser, so the partial transcript is available before the confirmation window expires.

• Backchannels should not interrupt. “Mm-hm”, “yeah”, “right”, “OK” are listening noises, not turn claims. If the partial matches a short backchannel list and the caller has stopped, un-duck and carry on.
• Stop words should interrupt instantly. “Stop”, “wait”, “no”, “hold on”, “operator”, “agent”. Bypass the confirmation window entirely for these; a caller who says “stop” and is talked over for another 300 ms has already formed their opinion.
• Some utterances should not be interruptible. A legal disclosure, a recording notice, a two-factor code being read out. Mark those explicitly and let barge-in queue rather than cancel.
• Never barge-in on the first 300 ms of your own utterance. That window is where echo from the previous turn and the caller’s trailing syllables land, and cancelling there produces the failure where the agent starts and immediately stops, repeatedly.

Testing it

Barge-in cannot be tested with unit tests alone, because the problems live in the acoustics. A small, fixed test matrix catches nearly everything:

1. Handset, mobile network, caller interrupts mid-sentence. The baseline. Measure how long the agent keeps talking after the first syllable of the interruption; that is your true stop latency, including the buffers you cannot flush.
2. Speakerphone in a reverberant room. The half-duplex case. If barge-in works on a headset and not here, your threshold is calibrated on the wrong signal.
3. Silence with the agent talking, for two minutes. There must be zero false interruptions. Any at all means echo is reaching your VAD and the suppression window is wrong.
4. A cough and a door slam. Neither may cancel.
5. Backchannel while the agent explains something long. “Mm-hm” must not stop it.
6. Interrupt twice in three seconds. This is where state machines that were written as flags fall into speaking and listening simultaneously.

Record every one of these as a stereo file with the two directions on separate channels. It is the only way to see what actually happened, and it makes the stop latency a measurement rather than an impression.