LibriSpeech, FLEURS and Common Voice

Almost every headline speech recognition number comes from read speech: someone in a quiet room reading a prepared sentence. Real audio is conversational, overlapping, accented and noisy, and word error rate rises steeply across that gap. A second, larger and less obvious lever sits between the transcript and the score: the text normaliser.

Word error rate, computed

WER is the minimum-edit distance between the reference transcript and the hypothesis, at word level, divided by the number of reference words.

          S + D + I
WER  =  -------------
              N

  S = substitutions, D = deletions, I = insertions
  N = number of words in the REFERENCE (not the hypothesis)

Reference:  i think we should probably just cancel the meeting tomorrow   (N = 10)
Hypothesis: i think we should probably just council the meeting to morrow

  "cancel"   -> "council"     1 substitution
  "tomorrow" -> "to morrow"   1 substitution + 1 insertion

  S = 2, D = 0, I = 1
  WER = (2 + 0 + 1) / 10 = 0.30 = 30%

Two properties of the formula surprise people. WER can exceed 100 per cent, because insertions are unbounded and the denominator is not: a system that hallucinates a paragraph over a three-word utterance can score 400 per cent. And WER weights every word equally, so getting “the” wrong costs the same as getting a drug name or an account number wrong — which is almost never true of the downstream use.

The normaliser is part of the metric

Before WER is computed, both strings are normalised. Which normalisations are applied is a scoring decision, it is rarely stated, and it moves the number more than most model choices do.

Reference (as spoken): "well, i paid Dr. Smith twenty-five dollars on Jan 3rd — didn't I?"
Hypothesis:            "Well I paid Doctor Smith $25 on January 3rd, didn't I"

Scored with NO normalisation, word by word:
  case differs, punctuation differs, "Dr." vs "Doctor",
  "twenty-five dollars" (3 tokens) vs "$25" (1 token),
  "Jan" vs "January"
  -> roughly 6 errors out of 13 reference words -> WER ~46%

Scored with a typical English normaliser
  (lowercase; strip punctuation; expand abbreviations;
   convert spelled numbers and currency to a canonical form;
   expand contractions):
  reference  -> well i paid doctor smith 25 dollars on january 3rd did not i
  hypothesis -> well i paid doctor smith 25 dollars on january 3rd did not i
  -> WER 0%

Same audio. Same model. Same transcript. 46 points apart.

This is not a contrived example; it is the ordinary case for any audio containing numbers, dates, currency, abbreviations or names. It is why published WERs from different sources are frequently incomparable, and why a system that appears to beat another by a couple of points may simply have been scored more leniently.

• Some normalisers are far more aggressive than others. The normaliser released alongside a widely used open speech model does a great deal of work — number and currency canonicalisation, abbreviation expansion, filler-word removal — and results produced with it are systematically lower than results produced with a minimal one.
• Filler removal is the biggest single lever on conversational audio. Whether “um”, “uh” and false starts are deleted from both sides before scoring can change a conversational WER by many points, because spontaneous speech is full of them.
• The rule for reporting is simple. State the normaliser by name and version, or publish it. A WER without one is not reproducible, and this is field two of the scoring-rule requirement.

The read-speech benchmarks

All three are read speech, and that is the shared limitation. A person reading a sentence produces planned syntax, no disfluencies, no overlapping speakers, no crosstalk, complete sentences, and a consistent microphone distance. Almost none of that is true of a phone call, a meeting or a voice interface used in a car.

Spontaneous speech, and why WER jumps

The corpora that test spontaneous speech are older and less quoted: conversational telephone collections, multi-microphone meeting recordings, noisy multi-speaker challenge sets, and long-form corpora built from earnings calls and podcasts. Systems score substantially worse on all of them, and the reasons decompose cleanly.

The size of the jump depends on which pair you compare and which normaliser you use, so a single multiplier would be dishonest. The reliable statement is directional and it holds in essentially every published table: within LibriSpeech alone, the “other” test split scores materially worse than the “clean” one for the same system, and moving from read speech to conversational telephone speech is a much larger step than that. If you are budgeting for a voice product, assume your real-world WER is several times the benchmark figure until you have measured it, on your own audio, with your own normaliser.

What WER does not count

• Which words were wrong. The one metric treats function words and content words identically. For most applications the useful measure is entity error rate over the terms that matter — names, numbers, product codes — and it is easy to compute alongside WER once you have annotated them.
• Who said it. Speaker attribution is a separate task with a separate metric, diarisation error rate. A transcript with perfect words and swapped speakers is useless for a meeting summary.
• Timing. Word-level alignment quality does not appear in WER at all, and it is what captions and transcript search depend on. A transcript with a perfect WER and drifting timestamps fails both.
• Readability. Casing, punctuation and paragraphing are usually normalised away before scoring, so a system that produces none of them can post an excellent WER and an unusable transcript.
• Latency. Streaming and batch recognition are different products, and a batch WER says nothing about how a streaming system behaves — the full latency budget covers the constraint.

Benchmarking on your own audio

Speech is one of the areas where a private benchmark is easiest to build and pays back fastest, because your audio differs from the public corpora along every axis that matters.

1. Collect two to three hours of real audio from your actual channel — the same codec, the same microphones, the same room, the same speakers. Two hours is enough to separate systems that differ meaningfully.
2. Have it transcribed by people, and write down your transcription conventions before they start: fillers in or out, numbers as digits or words, how overlap is marked. These decisions become part of your metric.
3. Fix one normaliser and use it for every system you compare, forever. Publish it with your results internally. Changing it invalidates every previous number.
4. Report WER and a domain-entity error rate over your important terms. The second usually drives the product decision.
5. Report per-speaker or per-accent breakdowns where your sample allows. An aggregate WER hides a subgroup the system fails on, which is both a quality problem and a fairness one.