Speaker Diarisation: Who Said What

Diarisation answers “who spoke when” over a single mixed audio stream. It is a clustering problem, not a recognition problem, and the first question to ask is whether you have been handed one badly enough to need it.

First, check whether you need it

Diarisation is inference from a mixed signal. If the speakers were ever separate before somebody mixed them, take the separate versions instead. This is the highest-value paragraph on the page and it is missing from nearly every guide.

• Telephony. A two-leg call has two RTP streams — the caller and the callee — and most platforms will hand you both as separate channels or separate recordings. Transcribe each independently and the attribution is exact, permanently, with no model involved.
• Video conferencing. Most conferencing platforms can produce per-participant audio tracks rather than a single mix. Ask for them before building anything.
• Recorded meetings in a room. Here you genuinely need diarisation, and if you control the hardware, a microphone array gives you direction of arrival as an extra, very strong feature.

A stereo file with one speaker per channel needs a channel split and a merge on timestamps, which is twenty lines of code and cannot be wrong. A diarisation model on the mixdown of the same call will confuse speakers several times an hour. Do not buy an error rate you were given the option not to have.

The classical pipeline

Most production diarisation is still a pipeline rather than one model, because the pieces can be swapped and debugged independently.

1. Voice activity detection. Find the regions containing speech. Everything downstream only sees those, so a false negative here is a permanent deletion no later stage can recover. See voice activity detection.
2. Segmentation. Cut the speech regions into pieces short enough that each is likely to contain one speaker. Either uniform windows — commonly 1.5 s with a 0.75 s shift — or explicit change-point detection. Uniform is more robust; change-point gives better boundaries when it works.
3. Embedding. Map each segment to a fixed-length vector that depends on who is speaking and as little as possible on what they said.
4. Clustering. Group the vectors. The number of groups is the number of speakers, and you usually do not know it.
5. Resegmentation. Re-derive frame-level boundaries given the cluster assignments, typically with a hidden Markov model over speaker states. This is what turns coarse 1.5 s blocks into boundaries that land near the actual turn.

End-to-end neural diarisation replaces steps 2 to 5 with one model trained to emit per-frame activity for each speaker, using permutation-invariant training so that the arbitrary ordering of speaker labels does not penalise it. Its main structural advantage is that it can mark two speakers active in the same frame, which the clustering pipeline cannot express at all.

What a speaker embedding encodes

Speaker embeddings are borrowed wholesale from speaker verification — the task of deciding whether two recordings are the same person. The x-vector design (a time-delay network with statistics pooling) and its descendants such as ECAPA-TDNN are trained on a classification objective over thousands of identities, and the layer before the classifier becomes the embedding.

The useful mental model is that the vector encodes vocal tract shape and habitual speaking style, and is trained to ignore the words, the channel and the room. It never fully succeeds at the last two, which is where several failure modes come from: the same person on a mobile and on a headset can land further apart than two different people on the same headset. This is why length normalisation, cosine rather than Euclidean distance, and a per-recording calibration step all matter, and why a diarisation system tuned on broadcast audio degrades on call-centre audio.

Clustering, and the unknown speaker count

Two families dominate, and the choice is mostly about whether you know the speaker count.

If you do know the speaker count, pass it. A two-person interview diarised with the count fixed at two is a substantially easier problem than the same audio with the count free, and most implementations accept a minimum and maximum.

Diarisation error rate, and its two loopholes

DER is the standard metric and it is a time-weighted sum of three errors:

        false alarm  +  missed speech  +  speaker confusion
DER  =  ------------------------------------------------------
                  total reference speech time

  false alarm       time labelled as speech that was silence
  missed speech     time that was speech and was labelled silence
  speaker confusion time attributed to the wrong speaker

Everything is measured in seconds, not segments, so one long
mistake costs more than several short ones.

Two conventions in the scoring script move DER as much as the model does, and a published DER that does not state both is uninterpretable:

• The collar. A forgiveness window around every reference boundary — 0.25 s on each side is the long-standing convention from the NIST Rich Transcription evaluations. It exists because human annotators cannot place a turn boundary to the millisecond, so scoring the region around it measures annotation noise. It also, unavoidably, discards the hardest part of the problem. Scoring with no collar produces a much larger number for the same system.
• Whether overlap is scored. Historically, regions where two people speak at once were excluded from scoring. A clustering pipeline assigns exactly one speaker per frame, so including overlap guarantees it misses one of them — every second of overlap is a second of missed speech it cannot win. Overlap is typically a low single-digit percentage of meeting audio and much higher in argumentative or informal conversation, so this decision alone can shift DER by several points.

DIHARD-style evaluations score with no collar and with overlap included, which is why DER figures from that lineage look far worse than figures from older evaluations for systems of similar quality. If you are comparing two systems, score them yourself with one script and one set of conventions.

Where it breaks

• Overlapping speech. The structural limit of the clustering pipeline. Interruptions, backchannels and crosstalk are where most confusion time comes from, and they are exactly the moments that matter in a meeting.
• Short turns. “Mm-hm”, “yeah”, “right”. A 0.3 s segment produces an embedding dominated by noise. Most systems either drop these or attach them to whoever was speaking around them, and a transcript in which the listener never agrees with anything is the visible symptom.
• Similar voices. Same sex, similar age, similar accent, same channel. Embeddings genuinely place these close together, and no threshold separates them cleanly.
• Channel changes mid-recording. Somebody switches from speakerphone to handset and their embedding moves. The system reports a new speaker.
• Identity is not attached. Diarisation gives you “speaker 1” and “speaker 2”, not names, and the numbering is arbitrary and unstable across recordings. Mapping to real identities is speaker identification, a separate task requiring enrolled reference audio — and one that carries the biometric-data obligations discussed in recording and retention.

Practically: merge the diarisation output with word timestamps by overlap rather than by segment, so a word is attributed to whichever speaker was active for most of its duration. That merge is described in timestamps and alignment.