Streaming Transcription and Stable Partial Text

A streaming recogniser does not send you text. It sends you a sequence of hypotheses about the same audio, each of which may rewrite the last. Treat them as text and your interface flickers; treat them as hypotheses and the fix is thirty lines.

The mental model: hypotheses, not text

Every streaming ASR API produces two kinds of message, whatever it calls them. A partial (interim, non-final) is the current best guess at everything said since the last commit point, and it can change arbitrarily on the next message. A final (committed) covers a closed span of audio and will not change again.

A real sequence, from one utterance:

  t=0.42  partial  "I want"
  t=0.61  partial  "I want to"
  t=0.88  partial  "I want to buy"
  t=1.10  partial  "I want to buy tomatoes"       <-- wrong
  t=1.34  partial  "I want to buy two matinees"   <-- rewritten
  t=1.55  partial  "I want to buy two tickets"    <-- rewritten again
  t=1.90  final    "I want to buy two tickets."

The prefix "I want to buy" was stable from t=0.88 onwards.
Everything after it churned three times.

The revision is not a defect. The decoder is using right context: later audio disambiguates earlier audio, and a system that never revised would be a system that ignored the evidence. Your job on the client is to distinguish the part that has settled from the part that has not, and to show the difference.

Sending the audio

Almost every real-time transcription API uses a WebSocket: you open it with credentials and configuration, then push binary audio frames and read JSON messages. The configuration you send and the field names you read back are vendor-specific — take them from your provider’s docs, not from here — but the client shape is the same everywhere.

// stream.ts -- Node 18+, "ws" package.
// Reads 16 kHz mono 16-bit little-endian PCM on stdin and streams it.
//
//   ffmpeg -f pulse -i default -ar 16000 -ac 1 -f s16le - | node stream.js
//
// The URL, the auth header and the JSON message shape below are
// placeholders. Replace them with your provider's; everything else
// is provider-independent.

import WebSocket from "ws";

const SAMPLE_RATE = 16000;
const BYTES_PER_SAMPLE = 2;
const CHUNK_MS = 40;
const CHUNK_BYTES = (SAMPLE_RATE * BYTES_PER_SAMPLE * CHUNK_MS) / 1000; // 1280

const ws = new WebSocket(process.env.ASR_URL!, {
  headers: { Authorization: "Bearer " + process.env.ASR_KEY },
});

let pending = Buffer.alloc(0);
let open = false;

ws.on("open", () => {
  open = true;
  // Vendor-specific handshake. Check the field names.
  ws.send(JSON.stringify({
    sample_rate: SAMPLE_RATE,
    encoding: "linear16",
    interim_results: true,
  }));
});

process.stdin.on("data", (buf: Buffer) => {
  pending = Buffer.concat([pending, buf]);

  // Backpressure: if the socket is congested, drop nothing and
  // send nothing -- let the buffer grow, and fail loudly if it
  // grows past a few seconds. Silently dropping audio produces
  // transcripts that are subtly wrong and impossible to debug.
  if (pending.length > SAMPLE_RATE * BYTES_PER_SAMPLE * 5) {
    throw new Error("audio backlog exceeded 5s; the socket is not keeping up");
  }
  if (!open || ws.bufferedAmount > CHUNK_BYTES * 8) return;

  while (pending.length >= CHUNK_BYTES) {
    ws.send(pending.subarray(0, CHUNK_BYTES));
    pending = pending.subarray(CHUNK_BYTES);
  }
});

process.stdin.on("end", () => {
  if (pending.length) ws.send(pending);
  ws.send(JSON.stringify({ type: "CloseStream" })); // vendor-specific
});

Two decisions in there are worth stating explicitly. Chunks of 20–100 ms are the usual range: smaller wastes frames on WebSocket overhead, larger adds its own duration to every latency measurement downstream. And the backpressure branch throws rather than dropping audio, because a dropped chunk produces a transcript that is wrong in a way no test will catch — the words are gone and nothing reports it.

The stable prefix algorithm

Here is the piece that guides omit. Keep the last few partials; the longest common word prefix across them is the part that has stopped moving. Render that solidly and render the tail as provisional.

// stable.ts -- no dependencies.

export type Rendered = { stable: string; tentative: string };

export class StableTranscript {
  /** Text of finals, already committed and immutable. */
  private committed = "";
  /** The last N partial hypotheses for the current open segment. */
  private history: string[][] = [];

  constructor(private readonly window = 3) {}

  /** A partial arrived. Returns what to display now. */
  partial(text: string): Rendered {
    const words = text.trim().split(/\s+/).filter(Boolean);
    this.history.push(words);
    if (this.history.length > this.window) this.history.shift();

    const prefix = longestCommonPrefix(this.history);
    return {
      stable: join(this.committed, prefix.join(" ")),
      tentative: words.slice(prefix.length).join(" "),
    };
  }

  /** A final arrived. It supersedes every partial for this segment. */
  final(text: string): Rendered {
    this.committed = join(this.committed, text.trim());
    this.history = [];
    return { stable: this.committed, tentative: "" };
  }
}

function longestCommonPrefix(seqs: string[][]): string[] {
  if (seqs.length === 0) return [];
  if (seqs.length === 1) return seqs[0];
  const shortest = Math.min(...seqs.map((s) => s.length));
  const out: string[] = [];
  for (let i = 0; i < shortest; i++) {
    const w = seqs[0][i];
    if (!seqs.every((s) => s[i] === w)) break;
    out.push(w);
  }
  return out;
}

function join(a: string, b: string): string {
  if (!a) return b;
  if (!b) return a;
  return a + " " + b;
}

The window parameter is the whole trade-off, and it is worth understanding rather than tuning blindly. A window of 1 means every word is immediately “stable” and you are back to flicker. A window of 3 means a word must survive three consecutive hypotheses before it stops moving, which at typical partial rates costs a couple of hundred milliseconds of display lag and removes almost all visible churn. Larger windows keep more text provisional for longer, which looks hesitant.

If your provider returns a per-word stability or confidence score, use it instead — it is derived from the decoder’s own lattice and is strictly better information than agreement across snapshots. The algorithm above is the fallback for the majority of APIs that do not.

Rendering without flicker

• Two spans, one line. Stable text in the normal colour, tentative text at reduced opacity. Never in a different position or a different size — a layout shift is more distracting than a word change.
• Never animate the tentative span. Fades and typewriter effects on text that is about to be rewritten produce exactly the churn you are trying to remove.
• Reserve the height. Give the transcript region a minimum height of two or three lines from the start, so the page does not jump when the second line appears.
• Pin scroll to the bottom, but not against the user. Auto-scroll while the reader is at the bottom; stop the moment they scroll up, and show a “jump to live” control instead.
• Commit visually on finals. When a final arrives, the whole segment becomes stable in one step. That moment is the natural place to apply punctuation and casing if you are doing it client-side.

Acting on partials

Display is the easy consumer. The hard one is logic — starting a model call, matching an intent, filling a form field — because acting on a hypothesis that then gets rewritten means undoing something.

1. Read-only actions on the stable prefix, always. Prefetching, retrieval, autocomplete suggestions: safe, because the worst case is wasted work.
2. Speculative model calls on the endpoint, cancellable. When your endpointer fires, issue the model request against the latest partial immediately rather than waiting for the final. When the final lands, compare it to what you sent; if it differs materially, abort the in-flight request and reissue. This buys you the whole ASR finalisation window and costs an occasional wasted call.
3. Side effects only on finals. Sending a message, charging a card, transferring a call. There is no version of speculative execution that makes an irreversible action safe.

Measuring streaming latency

“Real time” is not a measurement. Three different latencies matter, they move independently, and a vendor comparison that reports one number has chosen which of them to show you.

All three are measurable without any vendor cooperation, because you control the audio you send. The technique is to put known events at known times into the stream:

1. Build a test file with short, distinct utterances separated by silence — single words work well — and write down the sample offset of each word’s onset and of the end of speech before each silence.
2. Stream it in real time, not as fast as the socket accepts it. Pacing matters: pushing a file at ten times real time produces latency figures that describe a batch job.
3. Stamp the arrival of every message against the same monotonic clock you used to start the stream. The audio position corresponding to each stamp is simply elapsed time from the start of streaming, which is why the pacing has to be right.
4. For each known word, record the first message containing it (partial), the first of three consecutive messages containing it unchanged (stability), and the final covering it. Subtract the known onset.
5. Report p50 and p95 for each of the three. Streaming latency is long-tailed — a single reconnection or one congested second produces an outlier — and a mean over a two-minute run tells you about the outlier rather than about the service.

Run the same file through every recogniser you are considering. It takes an hour, it is reproducible, and it answers a question that vendor documentation is structurally unable to.

The failures you will hit