What Local Whisper Doesn't Do: Speaker Diarization

“Who said that” is not a feature Whisper is missing. It is a question the model has no representation for, and no amount of prompting will produce one, because there is no token in its vocabulary that could carry the answer.

What Whisper emits

Whisper’s decoder generates from a vocabulary of ordinary text tokens plus a small set of special ones, and the special ones are the entire control surface of the model:

• <|startoftranscript|> — the sequence-start token that every decode begins from.
• A language token, one per supported language: <|en|>, <|de|>, <|yue|> and so on. Predicting this token is how language detection works; there is no separate classifier.
• A task token, either <|transcribe|> or <|translate|>.
• <|notimestamps|>, or a sequence of timestamp tokens quantised to 20 ms increments across the 30-second window.
• <|endoftext|>, and a few tokens used for the previous-context conditioning between windows.

That is the list. There is no speaker token, no channel token and no turn boundary. Since generation is a softmax over this vocabulary, the model literally cannot express “speaker 2” — the probability mass has nowhere to go. Asking it to label speakers in a prompt produces plain text that looks like labels and is generated by the same mechanism that generates any other plausible continuation.

Why there is no speaker token

The training objective explains it completely. Whisper was trained on audio paired with transcripts scraped at scale from the internet — 680,000 hours in the original paper, and over five million hours of weakly and pseudo-labelled audio by large-v3. The supervision signal is the transcript, and internet transcripts overwhelmingly do not carry speaker attribution. A model can only learn to predict what the labels contain.

The input side reinforces it. Whisper’s front end takes a single mono channel at 16 kHz. Multi-channel recordings, where each speaker might have their own microphone, are downmixed before the model ever sees them, so even the one physical cue that would make separation trivial is destroyed at the boundary. This is worth knowing when your source really is a two-channel call recording: split the channels and transcribe each separately rather than downmixing, because that gives you perfect diarization for free and you will never get it back afterwards.

There is a subtler point about the representation itself. The encoder produces a representation optimised to predict what was said, and training pressure actively removes information that does not help with that — including much of the speaker-identity information present in the acoustics. The encoder output is not a good place to look for voice characteristics, which is why diarization systems run their own encoder over the raw audio rather than reusing Whisper’s.

What a diarization pass computes

Diarization is a separate pipeline with three stages, and pyannote.audio is the standard open implementation. The pyannote/speaker-diarization-3.1 model card documents the pipeline; it is MIT licensed, expects mono audio at 16 kHz, and as of 3.1 runs speaker segmentation and embedding in pure PyTorch.

• Segmentation. A neural model over short windows that predicts, for each frame, which of a small number of local speakers is active — including the case where two are active at once. This is where overlapped speech is detected, and it is the stage Whisper has no equivalent of.
• Speaker embedding. Each detected turn is encoded into a fixed-length vector by a model trained on speaker verification, so that two turns by the same person land close together regardless of what was said. This is a different training objective from ASR and it is why a second model is required.
• Clustering. The turn embeddings are clustered into speakers. If you know the number of participants, pass it; if you do not, the pipeline estimates it, and estimating it is the single largest source of error in the whole system.

The pipeline is gated on Hugging Face: you must accept the conditions for both pyannote/segmentation-3.0 and pyannote/speaker-diarization-3.1 with your account and pass an access token when loading. That is a licensing condition on the weights, not a technical obstacle to work around, and an offline deployment needs the weights fetched once by an account that has accepted it.

Joining the two, which is the hard part

You now have two independent timelines: Whisper’s text with timestamps, and the diarizer’s speaker turns with timestamps. Joining them sounds like an interval overlap and is not, for a specific reason.

Whisper’s segment timestamps are coarse. They are predicted tokens quantised to 20 ms, they describe segments that are often several seconds long, and the model is known to place them approximately — a segment boundary can sit a second away from where the speech actually stopped. A speaker change frequently happens in the middle of a Whisper segment, so assigning one speaker per segment mislabels every interruption.

The fix is word-level timestamps. faster-whisper takes word_timestamps=True, which uses cross-attention weights and dynamic time warping to place each word; WhisperX instead runs a separate forced-alignment model over the transcript, which is more accurate and adds another model to your deployment. With word times in hand, the join is: for each word, find the speaker turn covering its midpoint, then merge consecutive words with the same speaker into a labelled block.

segments, _ = model.transcribe("meeting.wav", word_timestamps=True)

for seg in segments:
    for w in seg.words:
        mid = (w.start + w.end) / 2
        speaker = turn_covering(mid)   # from the diarization output
        print(speaker, w.word)

Where it breaks

• Overlapped speech. The diarizer can mark two speakers active simultaneously. Whisper produces one linear transcript and will usually render the louder voice, so the words of the quieter one are simply absent and there is nothing to attribute.
• Speaker count. Clustering without a known count merges two similar voices or splits one person across two labels. The second failure is worse in practice because it is less obvious: a transcript with five speakers in a four-person meeting reads plausibly.
• Backchannels. “mm-hm”, “right”, a short laugh. Too short for a reliable speaker embedding, and often dropped by Whisper entirely, so the two systems disagree about whether anything happened at all.
• Compounding. The errors multiply rather than averaging. A word transcribed correctly but attributed to the wrong speaker is wrong; a word attributed correctly but transcribed wrong is wrong. Evaluate the joined output, not the two components.

None of this makes diarization unusable — it makes it a second system with its own accuracy budget, chosen and evaluated separately from whichever Whisper size you settled on.