Whisper's Architecture and Its Known Failure Modes

Whisper is an encoder-decoder transformer that reads exactly thirty seconds of log-mel spectrogram and writes a token sequence. Two design choices in that sentence — the fixed window and the unconstrained decoder — explain almost every surprising thing it does.

The architecture in one paragraph

OpenAI released Whisper in 2022. It is a standard transformer encoder-decoder. The encoder takes a log-mel spectrogram of a 30-second audio window, passes it through two convolution layers (the second with stride 2) and then a stack of transformer blocks. The decoder is an ordinary autoregressive transformer that cross-attends to the encoder output and emits byte-pair tokens. The original family used 80 mel bins; large-v3 uses 128. There is nothing novel in the architecture at all, which is the point the paper makes: the result came from the scale and breadth of weakly supervised audio-transcript pairs collected from the web, not from a new block.

It is published at several sizes — tiny, base, small, medium, large (revised as v2 and v3), and a distilled turbo variant with fewer decoder layers. Accuracy improves with size and varies enormously by language and by audio condition, so a single word error rate per size is not a meaningful figure and none is quoted here; measure on your own audio with a WER harness and a normaliser you have chosen deliberately.

The 30-second window is load-bearing

The encoder does not accept variable-length input. Every request is padded or trimmed to exactly 30 seconds before it reaches the model.

Any input audio
  -> resample to 16,000 Hz mono
  -> log-mel spectrogram, 10 ms hop
  -> pad with zeros (or trim) to exactly 3,000 frames
  -> conv stride 2 -> 1,500 encoder positions

Consequences that follow directly:

  a 2-second clip   = 2 s of speech + 28 s of digital silence
  a 45-second clip  = two windows, processed sequentially
  a 3-second clip and a 29-second clip cost the same to encode

Audio longer than the window is handled outside the model, by a loop that transcribes one window, uses the emitted timestamps to decide where the next window starts, and — by default in the reference implementation — feeds the previous window’s text back in as a prompt for continuity. Both halves of that loop are sources of trouble: the window boundary is chosen from timestamps the model produced itself, and the text conditioning carries any error forward.

It is a multitask model, controlled by tokens

Whisper does not have separate models for transcription, translation and language identification. It has one, and the task is selected by the special tokens the decoder is primed with. The sequence looks like this:

<|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|> Hello there.<|endoftext|>
                       ^^^^^^ ^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
                       |      |             |
                       |      |             +-- omit timestamp tokens
                       |      +-- transcribe, or <|translate|> for X -> English
                       +-- language, or let the model predict it

With timestamps enabled, the decoder interleaves time tokens:
  <|0.00|> Hello there. <|1.42|><|1.42|> How are you?<|3.10|>

Those time tokens are emitted on a 20 ms grid, and they are
predictions like any other token -- not measurements.

Two things worth extracting from that. First, the translate task goes in one direction only: any language into English. There is no English-to-French mode, and asking for one produces confident nonsense. Second, the timestamps come out of the same softmax as the words. They are the model’s guess at when it said something, not the output of an aligner, which is why word-level alignment is usually done by a second model.

Why it writes captions over silence

The most reported behaviour is that Whisper, given silence, music, or background noise with no speech in it, emits plausible sentences — often subtitle boilerplate of the “thanks for watching” variety, sometimes a channel name, sometimes a repeated fragment of a real sentence. It is not random and it is not a bug in the sampler. Three properties combine, and all three are architectural.

1. The decoder has no alignment constraint. A CTC or transducer model emits blank when there is no evidence, and blanks collapse to nothing. Whisper’s decoder is a language model with cross-attention; at every step its job is to produce the most likely next token given the tokens so far and whatever the encoder gave it. Nothing in that objective can express “there was no audio here” except the end-of-text token, which is one option competing against a whole vocabulary.
2. The training data taught it what silence is followed by. The corpus was weakly supervised audio paired with transcripts scraped at scale, and a meaningful fraction of subtitle tracks contain text that does not correspond to the audio under it — translator credits, channel sign-offs, sponsor lines, boilerplate over intro music. The model saw many examples where a low-information audio window was paired with exactly that kind of text. It learned the correlation faithfully. The hallucinated captions are not invented; they are the highest-probability continuation for that acoustic condition in the data it was shown.
3. Padding manufactures the condition. Because every input is padded to 30 seconds, a short clip is mostly digital silence by construction. A one-second “yes” is 29 seconds of the exact input state that most reliably triggers the behaviour. This is why the failure is reported far more often by people transcribing short utterances from a voice agent than by people transcribing hour-long meetings.

Once you see it as a probability argument rather than a defect, the mitigation is obvious: never present the model with a window that is mostly silence, and check the signals it gives you about its own uncertainty.

The other three failure modes

• Repetition loops. The decoder enters a state where the most likely continuation of a phrase is that phrase, and greedy decoding follows it until the token limit. The reference implementation detects this indirectly by compressing the output text and rejecting a segment whose gzip compression ratio is suspiciously high — repeated text compresses far better than prose.
• Timestamp drift on long audio. The sequential window loop starts each window where the previous one said it finished. An error of half a second in one window’s final timestamp shifts everything after it, and errors accumulate rather than cancel. On a long recording, later timestamps can be wrong by seconds even when the words are right.
• Language misdetection. Language is predicted from the first window. If that window is noise, music, or a greeting that exists in several languages, the wrong language token is chosen and the rest of the file is transcribed — or silently translated — under that assumption. On a phone call whose first two seconds are hold music, this is common. Pass the language explicitly whenever you know it.

What actually helps

The reference implementation exposes decoding controls for most of this. Names differ between the original package, the faster reimplementations and hosted APIs, so check them against the version you are running rather than copying blindly: