Running Whisper Locally With mlx-whisper

mlx-whisper is an MLX port of OpenAI’s Whisper that runs on the Mac’s GPU and never opens a socket after the weights are cached. For anything involving recordings you would rather not upload, that property is the whole point.

Install, including ffmpeg

1. Install ffmpeg. The project’s own instructions are brew install ffmpeg, and this is not optional — decoding anything that is not already a plain waveform goes through it, so a missing ffmpeg presents as a file-format error rather than as a missing dependency.
2. Install the package: pip install mlx-whisper, on a native arm64 Python, macOS 14.0 or newer.
3. Have an audio file to hand. Anything ffmpeg reads works — .mp3, .m4a, .wav, .flac, and the audio track of a video file.

One command to a transcript

mlx_whisper interview.m4a \
  --model mlx-community/whisper-large-v3-turbo \
  --output-format txt \
  --output-dir ./transcripts

With no --output-format the tool writes a text file beside the input; the flag also accepts subtitle and structured formats, which is what you want if the timings matter. --output-name overrides the derived filename. The command also reads standard input, so some-process | mlx_whisper - works for a piped stream.

The first run for a given model downloads the weights into the Hugging Face cache. After that the command is offline, which you can verify by running it with the network off.

The output format is worth choosing deliberately rather than taking the default, because it decides whether the timings survive. txt is the transcript with the timing thrown away. srt and vtt are subtitle files with per-segment start and end times, which is what you want if anything downstream has to point back at the audio. json keeps everything the model produced, including per-segment confidence signals, and is the right choice if a program rather than a person is the consumer. The related flags — --max-line-width, --max-line-count, --max-words-per-line and --highlight-words — only affect the subtitle formats.

From Python

import mlx_whisper

result = mlx_whisper.transcribe(
    "interview.m4a",
    path_or_hf_repo="mlx-community/whisper-large-v3-turbo",
    language="en",
    word_timestamps=True,
)

print(result["text"][:500])

for seg in result["segments"][:3]:
    print(f"[{seg['start']:6.2f} - {seg['end']:6.2f}] {seg['text']}")

The return value is a dictionary with the full text, a segments list carrying start and end times, and the detected language. With word_timestamps=True each segment also carries per-word timings, which costs extra decoding work and is the thing to turn off first if a long file is taking longer than you expected.

Passing language= explicitly is worth doing when you know it. Language detection runs on the opening window of audio, so a recording that begins with music, silence or a stray English greeting can be detected as the wrong language and then transcribed — or quietly translated — under that assumption for its whole length.

Two more arguments earn their place on real recordings. initial_prompt takes a short string that is prepended to the decoder’s context, and it is the supported way to hand the model vocabulary it would otherwise mangle — product names, people’s names, domain jargon. A sentence containing the terms, written in the style of the transcript, measurably changes how they come out; it is not a system prompt and long instructions in it do nothing useful. temperature defaults to a tuple rather than a scalar — (0.0, 0.2, 0.4, 0.6, 0.8, 1.0) — because the decoder retries a segment at successively higher temperatures when its own quality checks fail. Setting it to a single float disables that fallback, which makes decoding faster and makes bad segments stay bad.

Choosing a model size

OpenAI publishes the parameter counts for the Whisper family in the whisper repository’s README, and those are the honest inputs to a size decision:

tiny      39 M      ~1 GB   (OpenAI's stated VRAM guidance)
base      74 M      ~1 GB
small    244 M      ~2 GB
medium   769 M      ~5 GB
large  1,550 M     ~10 GB
turbo    809 M      ~6 GB

The same README describes turbo as an optimised version of large-v3 that transcribes faster with minimal degradation in accuracy, and states plainly that it is not trained for translation — it will return the original language even when a translate task is requested. For English transcription it is the sensible default; for putting non-English speech into English, OpenAI directs you to medium or large.

Nobody publishes a word error rate for these models on your audio, and none was measured here. What the parameter counts do tell you reliably is memory and relative work: tiny is forty times smaller than large and, being a bandwidth-bound decode like any other model on this hardware, correspondingly cheaper per second of audio. The way to choose is to run two sizes over five minutes of your own worst recording — background noise, accents, jargon — and read the difference.

Where it goes wrong

• Hallucinated text over silence. Whisper will confidently transcribe speech that is not there in long silent or near-silent passages, often a stock phrase from its training data. The--hallucination-silence-threshold option and --no-speech-threshold exist for this; the underlying tendency is a property of the model, not the port.
• Drift on long files. Transcription proceeds in thirty-second windows conditioned on the previous text, so one bad window can propagate. --condition-on-previous-text can be turned off, which trades coherence for isolation between windows.
• No speaker labels. Whisper transcribes; it does not diarize. Getting “who spoke” needs a separate model, and nothing in this package provides it.
• Memory is still one pool. The large models are small next to a language model, but they are competing for the same unified memory, so running a transcription and a 70B at once is subject to the same arithmetic as anything else here. See how MLX manages memory.

The wider treatment of the model family is in Whisper explained; this page is only the Apple silicon path to it.