Whisper Accuracy by Model Size, Sourced

There is no single OpenAI table of Whisper word error rates by model size. There are three separate published artefacts that people conflate into one, and the differences between them are exactly the differences that decide whether a number applies to you.

What is actually published

• The paper. Radford, Kim, Xu, Brockman, McLeavey and Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision”, submitted 6 December 2022. It reports zero-shot results across many datasets and its appendices carry per-size breakdowns. It predates large-v2, large-v3 and turbo entirely.
• The repository README. It carries a by-language performance breakdown, and it is specific about scope: the figure covers large-v3 and large-v2 only, evaluated on Common Voice 15 and Fleurs, using WER or, for languages where word boundaries do not carry the same meaning, character error rate. It has no per-size dimension at all.
• The individual model cards. Each openai/whisper-* card on Hugging Face includes an evaluation snippet and prints the WER it produces on LibriSpeech test-clean. This is the only place a per-size comparison exists under one methodology.

If a page shows you a five-row size-versus-WER table attributed to OpenAI, it has stitched together at least two of these, and the join is where the error is.

The per-size table, and its one dataset

Collecting the number each model card prints for LibriSpeech test-clean, with the Whisper text normalizer applied as the cards’ own snippets do:

model      params   WER on LibriSpeech test-clean
tiny        39M     7.547
base        74M     5.009
small      244M     3.432
medium     769M     2.900
large-v2  1550M     3.000

Sourced to the model cards published by OpenAI on Hugging Face at the time of writing — tiny, base, small, medium and large-v2 — all under the Apache-2.0 licence.

The shape is what you would expect for four of the five rows. Each step up in size roughly cuts the remaining error: 7.5 to 5.0 to 3.4 to 2.9. The returns diminish sharply — tiny to base saves 2.5 points, small to medium saves 0.5 — which is the practical argument for small being the sweet spot on clean English audio and for medium being the last size worth paying for on a CPU.

Why large scores worse than medium

The fifth row is the interesting one. large-v2, at twice the parameters, records 3.000 against medium’s 2.900. That is not a transcription error in this page and it is not a defect in the model.

LibriSpeech test-clean is read speech from public-domain audiobooks, recorded cleanly, by speakers reading prepared text. It is close to the easiest possible case for a speech recogniser, and models have been driven so far down it that the remaining error is largely annotation ambiguity. At WER under 3, differences of 0.1 are inside the noise of how the reference transcript spells things.

More importantly, this is not the axis Whisper was scaled along. The paper’s central claim is about robustness: zero-shot generalisation across accents, background noise, recording conditions and domains, without fine-tuning on any of them. Extra capacity buys you performance on hard, out-of-distribution audio. It buys you almost nothing on clean read speech because there was almost nothing left to buy. A benchmark that is saturated stops ranking models and starts ranking their idiosyncrasies.

The generalisable rule: a single-dataset WER table ranks models on that dataset. If your audio is a noisy phone call, a lecture hall, or a meeting with four people and a fan, the ordering above tells you very little, and the gap between medium and large will be the opposite sign and much larger.

By language, where the spread is enormous

The repository’s by-language breakdown covers large-v3 and large-v2 on Common Voice 15 and Fleurs, and its most useful property is its range rather than any individual value: the best-served languages sit in the low single digits and the worst-served are an order of magnitude higher. Whisper supports ninety-odd languages in the sense that it will emit text for them; it does not support them equally.

Two details in that figure are easy to miss and both change how you read it. Some rows are character error rate rather than word error rate, shown in italics, because word segmentation is not meaningful in those writing systems — a CER of 8 and a WER of 8 are not comparable quantities. And there is no per-size dimension, so the honest answer to “how much worse is small than large on Vietnamese?” is that OpenAI has not published it and you would have to measure it.

large-v3 also changed the front end: its model card records a switch from 80 to 128 Mel frequency bins and the addition of a Cantonese language token, trained on 1 million hours of weakly labelled and 4 million hours of pseudo-labelled audio. That is a different feature extractor, which is one more reason a v2 number does not transfer to v3.

Measuring it on your own audio

Thirty minutes of your own transcribed audio is worth more than every published table for choosing a size, and it is a short afternoon:

pip install jiwer transformers

python - <<'PY'
from jiwer import wer
from transformers.models.whisper.english_normalizer import EnglishTextNormalizer

norm = EnglishTextNormalizer({})
refs = [l.strip() for l in open("reference.txt")]
hyps = [l.strip() for l in open("whisper_small.txt")]
print(wer([norm(r) for r in refs], [norm(h) for h in hyps]))
PY

The normalizer is not optional if you want your number to be comparable with a published one. It lowercases, strips punctuation, expands contractions and standardises numbers, and every WER quoted on those model cards was computed after it ran. Comparing a raw WER against a normalised one will overstate your error by several points and send you buying a larger model for no reason.