Skip to content
Multigrid

Local models on Apple Silicon

MLX, unified memory and what a Mac can actually hold — where Apple Silicon wins for local inference and where it stops.

A Mac is an unusual machine to run a language model on. It has no discrete GPU and no video memory, so the question “will it fit?” has a different answer than it does on a graphics card: there is one pool of memory, the model competes with Safari for it, and the ceiling is set by a number macOS reports rather than by a number printed on a box. It also has an unusually large amount of memory bandwidth for its arithmetic throughput, which is the wrong trade for training and very nearly the right one for generating tokens one at a time.

These pages work from those two facts. MLX is Apple’s own array framework and the reason the software side is worth a separate cluster at all; unified memory is the reason the hardware side behaves the way it does. Where a page gives a number it says who published it and when, and where the number that matters is one nobody publishes — tokens per second for your model on your Mac — it derives the ceiling from arithmetic you can check and gives you the command that prints the real figure.

What MLX Is, and Why It Exists Alongside llama.cpp

Apple's array framework described by its two design decisions — a lazily evaluated graph and arrays that live in memory both processors can reach — rather than by its feature list.

8 min read

Converting a Hugging Face Model to MLX Format

Take a Hugging Face checkpoint through mlx_lm.convert and confirm it loads and generates, including what conversion actually changes on disk.

9 min read

Quantizing a Model for MLX

Quantize a checkpoint with mlx_lm.convert, verify the file-size change against the arithmetic that predicts it, and pick between the modes and mixed recipes.

9 min read

mlx-lm: Generating Tokens From Python, Not the CLI

Load a model, apply the chat template, stream tokens and control sampling and the KV cache through mlx-lm's Python API rather than its command-line wrapper.

9 min read

How MLX Manages Memory on Apple Silicon

The unified-memory allocation model, the buffer cache that makes MLX look like it is leaking, and the wired limit that decides how much of your RAM the GPU may use.

9 min read

Running an MLX Model as a Local OpenAI-Shaped API

Start mlx_lm.server, get an OpenAI-shaped completion back from localhost, and understand what the server does and does not do before you point anything at it.

9 min read

The mlx-community Repository on Hugging Face

How pre-converted MLX weights are named, what the suffixes mean, and how to tell a current conversion from an abandoned one when nothing is versioned.

8 min read

Running Whisper Locally With mlx-whisper

Transcribe a local audio file on Apple Silicon with no network call, and choose a model size from OpenAI's published parameter counts rather than by feel.

8 min read

Unified Memory Bandwidth and Tokens Per Second, Derived

Derives an upper bound on single-stream decode speed from a chip's published GB/s and a model's exact byte count, with every assumption named and the measurement command given.

10 min read

How Much RAM You Need to Run a 70B Model on a Mac

Builds the requirement from a real 70B checkpoint's exact byte count, the KV cache the architecture demands per token, and the working-set ceiling macOS actually reports.

10 min read

The Mac mini M4 for Local LLM Inference, by the Numbers

What Apple publishes for each Mac mini configuration, which quantized models those memory ceilings actually admit, and the decode ceiling each bandwidth figure implies.

9 min read

The Mac Studio's Memory Ceiling for Local LLMs, by the Numbers

Apple's published memory and bandwidth figures for each Mac Studio chip, and the largest quantized model each tier admits derived from real checkpoint byte counts.

9 min read

Why Apple Silicon Punches Above Its GPU Compute for Local LLMs

An argument that single-stream decoding is a memory-bandwidth problem, so the benchmark that ranks GPUs by arithmetic throughput is measuring the wrong thing.

9 min read

How Memory Bandwidth Has Changed Across Apple Silicon Generations

Apple's published GB/s figure for every M-series chip that has one, the two places the line went backwards, and what each change does to a fixed model's decode ceiling.

10 min read

Running Out of Memory on a Mac Running a Local Model

The exact Metal allocation errors MLX and llama.cpp print, what memory pressure and swap in Activity Monitor are telling you, and which of five fixes matches which cause.

10 min read

Choosing a Quantization Level for Your Mac's Memory Budget

Solve for bits per weight from the working set your Mac reports, the context you need and the parameter count you want, instead of applying a rule of thumb.

9 min read

Other topics