Word Error Rate: Measuring Transcription Properly

Word error rate is edit distance between the reference and the hypothesis, divided by the length of the reference. The formula is trivial. Everything that makes a WER number mean something happens before you apply it, in the normaliser — and the same transcript pair below scores 75% or 10% depending only on that.

The definition

         S + D + I
WER  =  -----------
         N

  S  substitutions   a reference word replaced by a different word
  D  deletions       a reference word missing from the hypothesis
  I  insertions      a hypothesis word with no reference word behind it
  N  words in the REFERENCE  (not the hypothesis)

Two identities that must hold for any valid alignment, and which
are the fastest way to check your own arithmetic:

  C + S + D = N                 (every reference word is matched,
                                 substituted or deleted)
  C + S + I = len(hypothesis)   (every hypothesis word is matched,
                                 substituted or inserted)

  where C = correct words.

S, D and I are not counted by eye. They come from the minimum edit distance between the two token sequences — the Levenshtein alignment, computed with a dynamic programming table where each cell costs 1 for a substitution, deletion or insertion and 0 for a match. That matters more than it sounds: the aligner finds the cheapest alignment, not the one a human would call semantically correct, and those differ often enough that hand-counting gives the wrong answer.

Note also that WER is not bounded above by 1. A hypothesis longer than the reference can produce more insertions than there are reference words. A model stuck in a repetition loop can easily score 400%.

Computed by hand, once

Take a sentence a human would call a perfectly good transcript.

REFERENCE   I'll meet you at 4:30 pm on Tuesday.
HYPOTHESIS  I will meet you at four thirty PM on Tuesdays

Tokenise on whitespace, change nothing else:

  ref (N = 8)  [ I'll | meet | you | at | 4:30 | pm | on | Tuesday. ]
  hyp (   10)  [ I | will | meet | you | at | four | thirty | PM | on | Tuesdays ]

Minimum-cost alignment:

  ref        hyp        operation
  ---------- ---------- ----------------
  I'll       I          substitution
  --         will       insertion
  meet       meet       correct
  you        you        correct
  at         at         correct
  4:30       four       substitution
  --         thirty     insertion
  pm         PM         substitution
  on         on         correct
  Tuesday.   Tuesdays   substitution

  S = 4    D = 0    I = 2    C = 4

Check:  C + S + D = 4 + 4 + 0 = 8  = N            OK
        C + S + I = 4 + 4 + 2 = 10 = len(hyp)     OK

        4 + 0 + 2      6
WER  =  ---------  =  ---  =  0.75  =  75%
            8          8

Seventy-five per cent word error rate on a transcript that is, to any reader, correct. Three of the four substitutions and both insertions are disagreements about writing conventions rather than about what was said. This is not a contrived example; it is what raw WER does to any transcript containing a time, a contraction and an abbreviation, which is most of them.

Where deletions come from

The pair above has no deletions, because the aligner found it cheaper to substitute than to delete. That is worth seeing explicitly, since the intuition “the model dropped a word, so that is a deletion” is often wrong. A deletion is only counted when there is no cheaper way to explain the missing word:

REFERENCE   send the report to marketing on friday      (N = 7)
HYPOTHESIS  send the report on friday                    (len 5)

  send    send      correct
  the     the       correct
  report  report    correct
  to      --        deletion
  marketing --      deletion
  on      on        correct
  friday  friday    correct

  S = 0   D = 2   I = 0   C = 5
  Check:  5 + 0 + 2 = 7 = N   OK
          5 + 0 + 0 = 5       OK

  WER = 2 / 7 = 0.286 = 28.6%

Deletions are the errors that matter most and the ones a WER number hides most effectively, because a dropped negation or a dropped digit costs the same 1 as a dropped “the”. Report S, D and I separately. A system with a high insertion rate and one with a high deletion rate at the same WER need completely different fixes, and treating them as equivalent is the most common mistake in ASR evaluation.

The same pair, scored four ways

Now take the original pair and change nothing except the normaliser applied to both sides before tokenisation. Each step is a decision somebody makes, usually silently.

Step 0 - raw, as computed above
  S=4 D=0 I=2  N=8
  WER = 6/8 = 75.0%

Step 1 - lowercase and strip punctuation, both sides
  ref  [ i'll meet you at 4:30 pm on tuesday ]        N = 8
  hyp  [ i will meet you at four thirty pm on tuesdays ]
  "pm" now matches "pm", so that substitution disappears.
  S=3 D=0 I=2
  WER = 5/8 = 62.5%

Step 2 - also expand contractions, both sides
  ref  [ i will meet you at 4:30 pm on tuesday ]      N = 9   <-- N changed
  hyp  [ i will meet you at four thirty pm on tuesdays ]
  S=2 D=0 I=1
  WER = 3/9 = 33.3%

Step 3 - also normalise numbers to a common spoken form
  ref  [ i will meet you at four thirty pm on tuesday ]   N = 10
  hyp  [ i will meet you at four thirty pm on tuesdays ]  len 10
  Only "tuesday" vs "tuesdays" remains.
  S=1 D=0 I=0
  WER = 1/10 = 10.0%

Same audio. Same model. Same output. 75.0% -> 10.0%.

Step 2 contains the subtlety that catches people: expanding I’ll to i will added a token to the reference, so the denominator went from 8 to 9. Normalisation does not only remove errors from the numerator, it changes N. Two normalisers that both look reasonable can therefore disagree in the direction you do not expect.

Four decisions produced a 7.5-fold difference, and none of them is obviously right or wrong. They are choices about what you are measuring:

Which is why a WER figure quoted without its normaliser is not a number. If you are comparing two vendors, run both outputs through your normaliser against your references. If you are reading a published comparison and it does not say what normalisation was applied, it has told you nothing.

An implementation with no dependencies

Short enough to read, so there is nothing hidden in it. This is the version used by the harness in the preprocessing page.

# wer.py -- word error rate with an explicit, visible normaliser.
# Python 3.9+, standard library only.

import re
from dataclasses import dataclass

CONTRACTIONS = {
    "i'll": "i will", "it's": "it is", "don't": "do not",
    "can't": "cannot", "won't": "will not", "i'm": "i am",
    "you're": "you are", "we'll": "we will", "that's": "that is",
}

def normalise(text, *, fold_case=True, strip_punct=True,
              expand_contractions=True, drop_fillers=False):
    """Every switch is a decision you are making on the record."""
    if fold_case:
        text = text.lower()
    if expand_contractions:
        for short, long in CONTRACTIONS.items():
            text = re.sub(r"\b" + re.escape(short) + r"\b", long, text)
    if strip_punct:
        text = re.sub(r"[^\w\s':]", " ", text)
    tokens = text.split()
    if drop_fillers:
        tokens = [t for t in tokens if t not in {"um", "uh", "erm", "mm"}]
    return tokens

@dataclass
class Result:
    substitutions: int
    deletions: int
    insertions: int
    correct: int
    reference_length: int

    @property
    def wer(self):
        if self.reference_length == 0:
            return float("nan")
        return (self.substitutions + self.deletions
                + self.insertions) / self.reference_length

def score(ref_tokens, hyp_tokens):
    n, m = len(ref_tokens), len(hyp_tokens)
    # d[i][j] = (cost, S, D, I, C) for ref[:i] against hyp[:j]
    d = [[(0, 0, 0, 0, 0)] * (m + 1) for _ in range(n + 1)]
    for i in range(1, n + 1):
        c, s, dd, ii, cc = d[i - 1][0]
        d[i][0] = (c + 1, s, dd + 1, ii, cc)
    for j in range(1, m + 1):
        c, s, dd, ii, cc = d[0][j - 1]
        d[0][j] = (c + 1, s, dd, ii + 1, cc)

    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if ref_tokens[i - 1] == hyp_tokens[j - 1]:
                c, s, dd, ii, cc = d[i - 1][j - 1]
                d[i][j] = (c, s, dd, ii, cc + 1)
                continue
            sub = d[i - 1][j - 1]
            dele = d[i - 1][j]
            ins = d[i][j - 1]
            best = min(sub, dele, ins, key=lambda t: t[0])
            if best is sub:
                c, s, dd, ii, cc = sub
                d[i][j] = (c + 1, s + 1, dd, ii, cc)
            elif best is dele:
                c, s, dd, ii, cc = dele
                d[i][j] = (c + 1, s, dd + 1, ii, cc)
            else:
                c, s, dd, ii, cc = ins
                d[i][j] = (c + 1, s, dd, ii + 1, cc)

    _, s, dd, ii, cc = d[n][m]
    return Result(s, dd, ii, cc, n)

if __name__ == "__main__":
    ref = "I'll meet you at 4:30 pm on Tuesday."
    hyp = "I will meet you at four thirty PM on Tuesdays"

    raw = score(ref.split(), hyp.split())
    print("raw          ", round(raw.wer * 100, 1), "%",
          "S", raw.substitutions, "D", raw.deletions, "I", raw.insertions)

    norm = score(normalise(ref), normalise(hyp))
    print("normalised   ", round(norm.wer * 100, 1), "%",
          "S", norm.substitutions, "D", norm.deletions, "I", norm.insertions)

Run it and you get the raw 75% and the step-2 33.3%. Number normalisation is deliberately left out of normalise: it is the one step that cannot be written generically, because the right canonical form depends on whether your product displays 4:30 or four thirty. Write it for your domain, apply it to both sides, and keep it in version control next to the eval set.

What WER does not measure

• Severity. “Do not administer” becoming “now administer” is two errors. So is “the report” becoming “a report”. If some words matter more, score those separately — keyword error rate or entity recall over the terms your product acts on.
• Speaker attribution. A perfect transcript with the speakers swapped scores 0% WER and is useless. Diarisation has its own metric and you need both.
• Timing. WER is indifferent to whether the words are in the right place in time.
• Distribution. A corpus-level WER averages over speakers. If it is 8% overall and 22% for one accent group, the average tells you nothing about the experience of that group; see the per-cohort method.
• Languages without word boundaries. Mandarin, Japanese and Thai are scored with character error rate instead, because “word” is a segmentation decision rather than a fact about the text.