Choosing a Whisper Model Size for Your VRAM Budget, Derived

OpenAI’s repository quotes ~10 GB of VRAM for Whisper large. The model’s weights at fp16 are 3.1 GB. Neither number is wrong, and the distance between them is the thing you actually need in order to choose a size for a card you own.

The inputs

Two sourced inputs, and nothing else. The first is the parameter count per size, which OpenAI publishes in the Whisper repository README: tiny 39M, base 74M, small 244M, medium 769M, turbo 809M and large 1550M. The same table gives the VRAM figures those models are said to require: ~1 GB, ~1 GB, ~2 GB, ~5 GB, ~6 GB and ~10 GB.

The second is bytes per weight, which is a property of the precision you load at, not of the model: 4 bytes at fp32, 2 at fp16 or bf16, 1 at int8. Whisper’s released checkpoints are fp16, and both the reference implementation and faster-whisper load at fp16 on a GPU by default.

Weights, which is the easy part

Weight footprint is parameters multiplied by bytes per weight. At fp16, that is:

tiny    39e6  x 2 =  78 MB
base    74e6  x 2 = 148 MB
small  244e6  x 2 = 488 MB
medium 769e6  x 2 = 1.54 GB
turbo  809e6  x 2 = 1.62 GB
large 1550e6  x 2 = 3.10 GB

Those are decimal gigabytes, and they are a derivation, not a measurement. The check that they are right is independent: the whisper.cpp README reports 2.9 GiB of disk for its unquantized large ggml file, and 2.9 GiB is 3.11 GB. A file on disk holding fp16 weights and nothing else agrees with the arithmetic to two significant figures, which is what you would expect if the arithmetic is the whole story for weights.

One caveat on that check: the ggml file also contains the vocabulary, the Mel filter bank and a header, so a few megabytes of it are not weights. At the scale of the large model that is inside the rounding; at tiny, where the arithmetic predicts 78 MB and the file is 75 MiB, the two agree only because the differences happen to run in opposite directions. Treat the agreement as confirmation of the method rather than as a precise identity.

At int8 halve every figure; large becomes 1.55 GB. The published faster-whisper benchmark supports the direction, reporting 2926 MB of VRAM for large-v2 at int8 against 4525 MB at fp16 — the difference is about 1.6 GB, which is the weight saving the arithmetic predicts, with the non-weight overhead unchanged.

The gap, which is the whole question

For large, OpenAI’s ~10 GB is 3.2x the 3.1 GB of weights. That multiplier is not a safety factor somebody invented. It has four named components, and knowing them tells you which ones you can turn off.

• The framework and the CUDA context. Importing PyTorch and initialising a CUDA context costs several hundred megabytes before a single weight is allocated, and PyTorch’s caching allocator holds on to freed blocks rather than returning them, so observed usage sits above live usage.
• Encoder activations. The encoder processes 1500 frames in one parallel pass across 32 layers at width 1280. Every intermediate tensor in that pass is live at once, and the feed-forward block widens to 5120 in the middle of each layer.
• The cross-attention KV cache. Fixed per window, and large enough to derive separately — see below.
• Beam search. Whisper’s default is beam size 5. Every per-hypothesis tensor — decoder self-attention cache, logits, the beam’s own bookkeeping — exists five times over. This is the multiplier most people forget, and it is the one you can remove for free by decoding greedily.

The cross-attention cache, derived

Whisper’s decoder cross-attends to the encoder output, and that output is the same for every token of a window, so both the keys and the values are computed once and cached. The size follows from the geometry:

32 decoder layers
x 2                (keys and values)
x 1500 frames      (30s of audio at Whisper's frame rate)
x 1280 width
x 2 bytes          (fp16)
= 245,760,000 bytes ~= 246 MB

At beam size 5 the decoder’s self-attention cache is replicated per hypothesis, and depending on implementation the cross-attention cache may be too, which puts a plausible ceiling near 1.2 GB for that structure alone. Assumptions: 32 decoder layers at width 1280 from the large architecture, 1500 encoder frames from the fixed 30-second window, fp16, and no memory-efficient attention kernel. Change any one of those and the number changes; the point is the order of magnitude, which is hundreds of megabytes rather than tens.

Notice what this implies about short audio. The cache is sized by the 30-second window, not by how much speech is in it, because Whisper pads every input to a full window. A three-second clip costs the same cross-attention memory as a thirty-second one.

Fitting a card

The practical procedure is to start from the weight arithmetic and then decide which of the four overheads applies to your runtime.

• Reference PyTorch implementation, beam 5. Use OpenAI’s quoted figures directly; they describe exactly this configuration. Large needs a 12 GB card to be comfortable, not a 10 GB one, because the quoted number is the requirement and not the headroom.
• faster-whisper. Its published table reports 4525 MB for large-v2 at fp16 and beam 5 — less than half OpenAI’s figure for the same model, because CTranslate2 does not carry the PyTorch allocator or its activation strategy. Batching pushes it back up: the same table shows 6090 MB at batch size 8.
• whisper.cpp. Its README reports ~3.9 GB for large, which is the fp16 weights plus about 0.8 GB. That is the cheapest configuration and it is cheapest because it does least.
• Quantized. Halving or quartering the weight term leaves the activation and cache terms untouched, so the total falls by less than the ratio suggests. Going from fp16 to int8 on large saves 1.55 GB of a 4.5 GB total in faster-whisper, not half of it.