Transcribing Audio Locally With whisper.cpp

whisper.cpp is a C/C++ port of Whisper with no Python, no PyTorch and no network access at runtime. That makes it the shortest path to a transcript on a machine you control, and it also makes every mistake a format mistake: it will refuse your audio, and it will happily load a quantized model whose quality you never chose deliberately.

Build it

The project is ggml-org/whisper.cpp, from the same authors as ggml and llama.cpp. It builds with CMake and has no required dependencies beyond a C++ toolchain.

git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
cmake -B build
cmake --build build -j --config Release

The binary you want lands at ./build/bin/whisper-cli. This is the single most common source of a stale command line: the CLI used to be called main and sat in the repository root, and a large amount of writing about whisper.cpp still says so. If a tutorial tells you to run ./main, it predates the rename and its flags may predate other things too.

Get a model

whisper.cpp does not read the original PyTorch checkpoints. It reads ggml-format files, and the repository ships a script that downloads converted ones:

sh ./models/download-ggml-model.sh base.en
sh ./models/download-ggml-model.sh large-v3

The whisper.cpp README publishes disk and memory figures for the unquantized ggml models: tiny at 75 MiB on disk and roughly 273 MB resident, base at 142 MiB and ~388 MB, small at 466 MiB and ~852 MB, medium at 1.5 GiB and ~2.1 GB, and large at 2.9 GiB and ~3.9 GB. Those are worth internalising because they are much closer to the raw weight size than the figures OpenAI quotes for the reference Python implementation — see where the difference between those two sets of numbers goes.

The .en variants exist for tiny, base, small and medium and are trained on English only. If your audio is definitely English they are the better choice at the same size, because none of the model’s capacity is spent on the other ninety-odd languages. There is no large.en; at that size OpenAI released multilingual only.

The audio format is not optional

whisper-cli reads 16-bit PCM WAV at 16 kHz, mono. This is not a limitation of the port. Whisper’s front end computes an 80-bin (or, for large-v3, 128-bin) log-Mel spectrogram from a 16 kHz signal, and every frequency bin’s meaning is defined relative to that sample rate. Feed it 44.1 kHz and the spectrogram no longer describes the same frequencies the model was trained on.

The reference Python implementation hides this by shelling out to ffmpeg for you. whisper-cli does not, so you do it yourself:

ffmpeg -i interview.m4a -ar 16000 -ac 1 -c:a pcm_s16le interview.wav

-ar 16000 resamples, -ac 1 downmixes to one channel and -c:a pcm_s16le writes signed 16-bit little-endian samples. The downmix is worth pausing on if your source is a two-channel recording with one speaker per channel: averaging the channels throws away the only clean speaker separation you had. Split the channels and transcribe each separately instead, because Whisper itself will not give it back to you — see why there is no speaker channel in the output.

Transcribe

./build/bin/whisper-cli \
  -m models/ggml-large-v3.bin \
  -f interview.wav \
  -l en \
  -t 8 \
  -osrt -otxt

-m is the model file, -f the audio, -l the language code (auto to let the model detect it, which costs an extra pass over the first window), and -t the thread count. -osrt and -otxt write interview.wav.srt and interview.wav.txt next to the input; there are equivalents for VTT, CSV, LRC and JSON.

Thread count deserves a moment. More threads help until the matrix multiplications become bound by memory bandwidth rather than arithmetic, and past that point they cost you through cache contention. Setting -t to the number of physical cores is the sensible default; setting it to the number of hardware threads usually is not, because two hyperthreads on one core share the same vector units. There is more on where that ceiling comes from in the CPU-only speed derivation.

Choosing a quantization level

whisper.cpp ships a quantizer, and the choice between its levels is arithmetic rather than taste:

./build/bin/quantize models/ggml-large-v3.bin \
  models/ggml-large-v3-q5_0.bin q5_0

The ggml block formats store weights in blocks of 32 with a shared scale, so the effective bits per weight is not the number in the name. A q5_0 block is 32 weights at 5 bits plus one fp16 scale: (32 x 5 + 16) / 32 = 5.5 bits per weight. A q8_0 block is 32 weights at 8 bits plus one fp16 scale: (32 x 8 + 16) / 32 = 8.5 bits per weight. The unquantized ggml files are fp16, at 16 bits per weight.

That gives you the file size before you download anything. Taking the README’s 2.9 GiB for large as the fp16 baseline: q8_0 should land near 2.9 x 8.5/16 = 1.54 GiB, and q5_0 near 2.9 x 5.5/16 = 1.0 GiB. Both figures are the arithmetic, not a measurement; the actual file differs slightly because a few tensors are usually left at higher precision. The point is the ratio, and the ratio is the whole decision: q5_0 buys you a large model in the memory footprint of a medium one.

What it costs is harder to state honestly. Quantization error is not uniform across a network — it concentrates in whichever weights have the widest dynamic range, and for a speech model the audible consequence is usually not gibberish but a higher rate of dropped words, invented punctuation and repetition loops on quiet passages. Nobody publishes a WER-versus-quantization-level table for whisper.cpp that would apply to your audio, so the honest procedure is to transcribe five minutes of your own material at fp16 and at q5_0 and diff the two transcripts. If they agree, you have your answer for that material; if they diverge on exactly the passages you care about, you have that answer too.

1. Clone and build with CMake; confirm ./build/bin/whisper-cli exists.
2. Download a model with sh ./models/download-ggml-model.sh large-v3.
3. Convert your audio: ffmpeg -i in.m4a -ar 16000 -ac 1 -c:a pcm_s16le out.wav.
4. Run whisper-cli with -m, -f, -l, -t and an output flag; read the generated .srt.
5. Quantize to q5_0, transcribe the same file again, and diff the two transcripts before you adopt the smaller model.