distil-whisper Explained

distil-whisper is not a smaller Whisper trained from scratch. It is Whisper large with almost all of its decoder deleted and the remainder retrained to imitate the original. Understanding which half was cut explains both why it is so much faster than its size suggests and where its accuracy gives way.

The distillation recipe

Whisper is an encoder-decoder transformer. The encoder reads a 30-second log-Mel spectrogram and produces a fixed sequence of audio representations; the decoder generates text tokens one at a time, attending to its own output so far and, through cross-attention, to the encoder’s output.

Hugging Face’s distil-whisper keeps the encoder unchanged and frozen, copied wholesale from the teacher. The decoder is cut to two layers, initialised from the teacher’s first and last decoder layers — for distil-large-v3 that is teacher layers 1 and 32. The student is then trained on a knowledge-distillation objective: the KL divergence between the student’s and the teacher’s output distributions, plus a cross-entropy loss against pseudo-labels the teacher generated over unlabelled audio. Hugging Face documents the recipe in the distil-whisper repository, and its model card records 22,000 hours of audio drawn from nine permissively licensed open datasets.

The choice of first-and-last is not arbitrary. A transformer decoder’s first layer does most of the work of turning token embeddings into something the rest of the stack can use, and its last layer does most of the work of shaping the final hidden state into something the output projection can read. Keeping both preserves the two interfaces and throws away the middle, where the representation is refined rather than transformed.

Where the parameters went

Whisper large is 1550M parameters, as OpenAI’s repository states, and distil-large-v3 is 756M, as its model card states. That reduction is fully explained by the decoder cut, and the arithmetic is worth doing because it shows the encoder is untouched.

Whisper large has 32 encoder layers and 32 decoder layers at a model width of 1280, with a feed-forward width of 4x that. Counting only the weight matrices, an encoder layer holds four d x d projections for self-attention and two d x 4d matrices for the feed-forward block:

encoder layer = 4d^2 + 8d^2 = 12d^2
             = 12 x 1280^2 = 19.7M
32 layers    = 629M

decoder layer = self-attn 4d^2 + cross-attn 4d^2 + ffn 8d^2 = 16d^2
              = 16 x 1280^2 = 26.2M
32 layers     = 839M

Cutting the decoder from 32 layers to 2 removes 30 x 26.2M = 786M parameters. Subtracting that from the published 1550M gives 764M — against a published 756M, a 1% gap that is accounted for by biases, layer norms and the exact treatment of the output projection, none of which this arithmetic counts. The assumptions are: layer counts and width taken from the large architecture, feed-forward expansion of 4, and weight matrices only.

So roughly 83% of what remains is the encoder. distil-whisper is, in parameter terms, Whisper large’s ears with a much simpler mouth.

Why latency falls further than size

The model card reports 6.3x relative latency for distil-large-v3 against large-v3, from a model that is 51% of the size. A 2x parameter reduction does not usually buy a 6x speedup, and the reason it does here is the asymmetry between the two halves.

The encoder runs once per 30-second window, over a fixed 1500-frame sequence, in one parallel pass. The decoder runs once per output token, sequentially, and each of those passes is bound by how fast the decoder’s weights can be read from memory rather than by arithmetic. A one-minute utterance might be two encoder passes and several hundred decoder passes. Cutting the encoder would save arithmetic on the small part of the bill; cutting the decoder from 32 layers to 2 cuts the per-token memory traffic by roughly sixteen on the part that repeats hundreds of times.

This is the same reason the KV cache dominates generation cost in text models, arriving from the other direction: the sequential half of an encoder-decoder model is the half whose cost multiplies by output length.

What the published numbers say it costs

The distil-large-v3 model card publishes a comparison against large-v3, and these are the figures at the time of writing rather than anything measured here:

• large-v3 — 1550M parameters, relative latency 1.0, short-form WER 8.4, sequential long-form WER 10.0, chunked long-form WER 11.0.
• distil-large-v3 — 756M parameters, relative latency 6.3x, short-form WER 9.7, sequential long-form WER 10.8, chunked long-form WER 10.9.
• distil-large-v2 — same 756M, 5.8x, short-form 10.1, but sequential long-form 15.6, which is where v3’s training change shows.

The shape of that table is the interesting part. On short-form audio the distilled model gives up about 1.3 WER points. On long-form audio it gives up 0.8 sequentially and is very slightly better chunked. Its predecessor was 5.6 points worse on sequential long-form, and the model card attributes the fix to training on samples packed to the full 30 seconds rather than on 7-second averages — the earlier model had effectively never seen a full window during distillation and fell apart when asked to condition on one.

When it is the wrong choice

distil-large-v3 is English-only. The distillation was run on English pseudo-labels, and the two surviving decoder layers have no capacity set aside for anything else. If your audio is multilingual, or you need Whisper’s translate task, this is the wrong model and the right comparison is against a smaller multilingual Whisper instead.

The second caution is about rare tokens. A two-layer decoder has less room to model the joint distribution over unusual sequences — surnames, product codes, jargon, spelled-out identifiers. Word error rate averaged over a benchmark hides this, because rare tokens are rare. If the entire point of your transcription is to catch the part-number somebody read out, evaluate on that and not on WER.

Both models carry the MIT licence they inherit from OpenAI’s Whisper, so neither poses a redistribution problem. The practical decision is between a distilled English model at 6x and a smaller multilingual Whisper at whatever its own size buys you, and the per-size accuracy numbers are the other half of that decision.