The Latency Budget for Real-Time Local Whisper Transcription

A live transcription system has to finish each chunk before the next one arrives, or its lag grows without bound. The awkward part of Whisper is that halving the chunk does not halve the work, so the usual instinct — shorter chunks for lower latency — makes the system harder to sustain rather than easier.

What the budget is made of

End-to-end delay between a word being spoken and appearing on screen is a sum of four terms, and only one of them is the model:

• Capture. You cannot transcribe a chunk until it has happened. A 5-second chunk contributes 5 seconds of delay for its first word and 0 for its last, so it contributes 2.5 seconds on average and 5 in the worst case. This term is pure arithmetic and no amount of hardware touches it.
• Inference. Encoder pass plus autoregressive decode. The subject of the rest of this page.
• Queueing. Zero if inference finishes before the next chunk lands, unbounded if it does not. There is no middle ground over time; a system that is 5% too slow is fine for a minute and thirty seconds behind after ten.
• Post-processing. Deduplicating overlapped text, punctuation, and any downstream call. Small, but it is inside the deadline.

The 30-second window never shrinks

Whisper’s encoder does not take variable-length audio. Its input is a log-Mel spectrogram of exactly 30 seconds — 480,000 samples at 16 kHz, producing 3000 spectrogram frames that the convolutional front end halves to 1500 encoder positions. Shorter audio is zero-padded to that length before it reaches the model; longer audio is split.

This is architectural, not a default. The encoder’s learned positional embeddings are defined for 1500 positions and its self-attention is over all of them. A one-second clip therefore costs the same encoder forward pass as a full window: the same 1500 positions, the same attention matrices, the same arithmetic, with 29 seconds of it spent on silence you inserted.

The chunk arithmetic

Write E for the compute cost of one encoder pass over a window. If you transcribe in chunks of c seconds, you run one encoder pass every c seconds of wall clock, so the sustained compute rate is E/c per second. Normalising against the batch case where c = 30:

chunk c = 30s  ->  E/30   ->  1.0x the batch compute rate
chunk c = 10s  ->  E/10   ->  3.0x
chunk c = 5s   ->  E/5    ->  6.0x
chunk c = 2s   ->  E/2    ->  15.0x
chunk c = 1s   ->  E/1    ->  30.0x

A live system on 2-second chunks is doing fifteen times the encoder work per second of audio that the same model does on a batch file. That is the number that surprises people who assume a real-time transcriber is cheaper than a batch one because it handles less audio at a time.

The decoder behaves differently and mostly does not rescue you. Its cost scales with the number of tokens emitted, which does scale with how much speech there is, so decoding a 2-second chunk really is roughly a fifteenth of decoding a 30-second one. But the decoder also pays a fixed per-window prompt cost — the special tokens, the language token, the task token — and that is per chunk, not per second.

Finding your floor from one measurement

To sustain chunk size c you need one window’s inference to complete in less than c seconds. So the minimum sustainable chunk is simply the time your machine takes to process one 30-second window, plus whatever headroom you want.

You can read that number straight out of a published batch timing. The faster-whisper README reports large-v2 at fp16 with beam 5 transcribing 13 minutes of audio in 1 minute 3 seconds, on an NVIDIA RTX 3070 Ti 8GB with CUDA 12.4. SYSTRAN publishes both the figures and the hardware. Thirteen minutes is 26 windows of 30 seconds, so:

63 s / 26 windows = 2.4 s per 30-second window

On that hardware, with that model and that beam size, no chunk shorter than about 2.4 seconds can be sustained — not because the audio is too short but because the window is always 30 seconds long and takes 2.4 seconds to process regardless. Adding 40% headroom for jitter and post-processing puts a practical floor near 3.5 seconds. The assumptions are that the published figure is representative, that per-window cost is uniform across the file, and that a padded window costs the same as a full one, which is exactly what the previous section argues.

Two levers move the floor down and both are visible in the same published table. Dropping to int8 gives 59 seconds for the same work (2.3 s per window, barely different, because the bottleneck at beam 5 is not weight bandwidth). Dropping the model size is the real lever: the same arithmetic applied to a small model rather than large-v2 changes the per-window cost by roughly the ratio of their parameter counts, which is a different derivation covered in the CPU-only speed page.

Overlap, and why you cannot avoid paying for it

Chunking at fixed boundaries cuts words in half, and a model asked to transcribe half a word will invent the other half. The standard fix is to overlap consecutive chunks by a second or two and deduplicate the transcripts where they agree. That works, and it raises the compute rate again by the overlap fraction: 5-second chunks with 1 second of overlap advance the stream by 4 seconds per window, so the rate is E/4 rather than E/5.

The alternative is to let voice-activity detection choose the boundaries, cutting during silence rather than on a timer. This is strictly better for accuracy — no word is ever split — and strictly worse for latency predictability, because a speaker who does not pause for eleven seconds gets an eleven-second chunk and eleven seconds of delay. Most production systems run both: VAD boundaries with a hard timeout that forces a cut, accepting a split word at the timeout rather than an unbounded wait.

The last thing worth budgeting for is the failure mode rather than the latency. Whisper fed near-silence tends to emit text it associates with silence in its training data, which is why streaming transcripts of quiet rooms fill up with subtitle credits and thank-yous. A VAD gate in front of the model is the fix, and it is also the largest single saving available on sparse audio, since silent windows cost a full encoder pass to transcribe into nothing.