whisper.cpp Hallucinating Text on Silence

A recording with thirty seconds of room tone transcribes as a thank-you for watching, a subtitle credit, or the same phrase repeated a dozen times. This is a known and well-documented failure mode of the Whisper models themselves, and there is a specific reason the obvious knobs do not fix it.

What the output looks like

The invented text has a recognisable character. It tends to be subtitle-corpus boilerplate — thanks for watching, subtitles by somebody, a channel sign-off — or a short phrase looping until the segment ends. Timestamps are usually plausible and the segments are well-formed. Nothing about the output marks it as fabricated, which is what makes it dangerous in an automated pipeline.

It clusters at particular places: leading and trailing silence, gaps between speakers, and stretches of steady non-speech noise like air conditioning or traffic. Long files produce more of it than short ones because they contain more such stretches.

This is not specific to whisper.cpp. It is a property of the OpenAI Whisper models, reproduced across implementations, and it has been studied directly — there is a body of published work on Whisper hallucination induced by non-speech audio, including an investigation of hallucinations induced by non-speech audio. A different runtime will not remove it.

Why silence produces sentences

Whisper is a sequence-to-sequence model trained on a large corpus of audio paired with transcripts harvested at scale. That corpus contains segments where the text does not correspond to the audio — subtitle files with boilerplate at the top and tail, credits over music, text that continues past the speech. The model therefore learned real associations between non-speech audio and fluent text.

The decoder is also autoregressive and conditioned on the preceding context. Given a 30-second window with nothing to transcribe, it still produces a distribution over tokens, and the highest-probability continuation is whatever the training data most often placed there. The model has no representation for “nothing was said” other than the special no-speech token, and the boilerplate associations compete with it directly.

Repetition loops have a related origin. Once a phrase is emitted, it becomes context, and the conditional probability of repeating it rises. With no acoustic evidence pulling the decoder anywhere else, the loop is stable until the window ends.

Why the built-in thresholds miss it

Whisper ships with a filtering mechanism: a no-speech probability from a dedicated token, and an average log-probability across the generated tokens. A segment is suppressed when the first is high and the second is low. whisper.cpp exposes both, along with an entropy threshold, as command-line flags:

-nth N,    --no-speech-thold N     # no-speech probability threshold
-lpt N,    --logprob-thold N      # average log probability threshold
-et  N,    --entropy-thold N      # entropy threshold for decoder fallback
-nf,       --no-fallback          # do not use temperature fallback
-sns,      --suppress-nst         # suppress non-speech tokens

The reason tuning these disappoints is structural. The filter assumes hallucinated output is low-confidence output. Hallucinated boilerplate is not: it is high-probability text, because it is exactly what the training data associated with this audio. Its average log-probability looks like that of a correct transcription, so it passes the very check designed to catch it. Raising the aggressiveness of the thresholds starts discarding quiet real speech before it reliably discards confident invention.

--suppress-nst and the blank-suppression behaviour help with a narrower problem — onomatopoeia, bracketed sound descriptions, blank segments — and are worth having. They do not address a fluent sentence.

The flags that help: VAD first

The intervention that works is to not send silence to the model at all. If the audio reaching the encoder contains speech, the failure mode has nothing to trigger it. whisper.cpp has built-in voice activity detection using a Silero VAD model, driven by these flags, documented in the whisper.cpp repository README:

--vad                                    # enable VAD
-vm FNAME, --vad-model FNAME             # path to the VAD model
-vt  N,    --vad-threshold N             # speech probability threshold
-vspd N,   --vad-min-speech-duration-ms  N
-vsd  N,   --vad-min-silence-duration-ms N
-vmsd N,   --vad-max-speech-duration-s   N
-vp   N,   --vad-speech-pad-ms           N
-vo   N,   --vad-samples-overlap         N

# typical invocation
whisper-cli -m models/ggml-large-v3-turbo.bin \
  --vad -vm models/ggml-silero-v5.1.2.bin \
  -f audio.wav

The VAD model is a separate download from the Whisper weights, and omitting -vm is the usual reason --vad appears to do nothing. Two of the tuning parameters matter more than the rest: --vad-speech-pad-ms adds audio either side of a detected segment, and setting it too low clips word onsets, which trades hallucination for truncation; --vad-min-silence-duration-ms decides how long a pause has to be before a segment is cut, and setting it too low fragments sentences in a way that costs accuracy because the decoder loses context.

Building a pipeline that does not do this

1. Segment with VAD before transcription, and transcribe only the speech regions. This is the single largest reduction available and everything else is secondary to it.
2. Normalise the input. Whisper expects 16 kHz mono; resampling artefacts and DC offset both make the VAD’s job harder. ffmpeg -i in.m4a -ar 16000 -ac 1 -c:a pcm_s16le out.wav is the standard preparation.
3. Disable cross-segment conditioning for long files if loops persist. Not carrying previous text into the next window removes the mechanism that sustains a repetition, at some cost to consistency of names and terminology across the file.
4. Post-filter on the known boilerplate. A blocklist of the specific phrases your corpus produces is inelegant and effective, because the invented text is drawn from a small set. Log what you remove rather than deleting silently, so you can tell if it ever starts removing real speech.
5. Sanity-check duration against text volume. A segment with several seconds of audio and a suspicious ratio of words to time is worth flagging for review; that check catches loops without needing to know the phrases in advance.

For the wider picture of running Whisper locally, including which model size to pick and how the pipeline fits together end to end, see the whisper.cpp walkthrough.