Running llama.cpp With the Metal Backend on Apple Silicon

On macOS the Metal backend is not something you switch on; it is on unless you removed it. The useful work is proving that it is actually executing the model, because a build that silently fell back to the CPU looks identical until you time it.

Building it

llama.cpp’s build documentation states it directly: on macOS, Metal is enabled by default, and to disable it at compile time you pass -DGGML_METAL=OFF. There is no positive flag to remember.

1. Install the toolchain. You need the Xcode command line tools for the Metal shader compiler and CMake for the build itself; a Homebrew cmake is fine.
2. Configure and build from the repository root:

   cmake -B build
   cmake --build build --config Release -j

3. Run something small and read the first thirty lines of the log rather than skipping to the output:

   ./build/bin/llama-cli -m models/model-Q4_K_M.gguf -ngl all -p "hi" -n 16

4. Confirm the Metal lines below are present. If they are not, you have a CPU build and everything after this point is measuring the wrong thing.

One build-time subtlety is where the compiled shaders live. llama.cpp can embed the Metal library into the binary or load a default.metallib from disk, and it says which it did: using embedded metal library for the first, loading ‘...’ naming a path for the second. A binary copied away from its build tree without its .metallib is the usual cause of a Metal backend that worked yesterday, and the fallback line, default.metallib not found, loading from source, is the one to search for when start-up suddenly takes seconds longer.

The lines that prove Metal loaded

Device initialisation prints a block from ggml_metal_device_init. The names and the exact set vary with the build, but the shape is fixed and each line is a claim you can check:

ggml_metal_device_init: GPU name:   <your chip>
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_device_init: simdgroup reduction   = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory    = true
ggml_metal_device_init: recommendedMaxWorkingSetSize  = <N> MB

simdgroup matrix mul. reading true is the one that matters for throughput: it says the device has the matrix instructions the fast kernels are written against. has unified memory is why the rest of this page differs from a discrete-GPU page. And recommendedMaxWorkingSetSize is the ceiling discussed below.

Absence of the block is the diagnosis. So is a later line reporting that layers were assigned to the CPU when you asked for all of them on the GPU — llama.cpp prints the offload counts at load, and those counts, not your flags, are what actually happened.

A controlled comparison

Logs prove the backend registered. To prove it is doing the work, run the same benchmark twice and change exactly one thing. llama-bench prints a backend column for precisely this, alongside a test column carrying pp (prompt processing) and tg (token generation) rows and a t/s column with a standard deviation over repeated runs.

# GPU path
./build/bin/llama-bench -m models/model-Q4_K_M.gguf -p 512 -n 128

# CPU control: same binary, no offload at all
./build/bin/llama-bench -m models/model-Q4_K_M.gguf -p 512 -n 128 --device none

Read the backend column first — it should say Metal in one run and CPU in the other — and only then the rates. The interesting part is that the two rows move differently. Prompt processing is arithmetic-bound and should improve a great deal on the GPU; token generation is memory-bandwidth-bound and improves much less, because on a unified-memory Mac both processors are reading the same RAM at similar speed. A build where pp barely moved is a build that is not really using Metal, whatever the log said.

The same two-run discipline is what to use on any backend — the Vulkan page applies it to a different set of log lines. There is no honest way to tell you what those numbers should be. They depend on the chip, the memory configuration, the model and the quant, and nobody publishes the combination you own. The comparison is the measurement; the absolute figure is only meaningful against your own earlier run.

-ngl on unified memory

-ngl takes an exact number, auto or all, and defaults to auto. On a discrete GPU the flag is a rationing decision between two separate memories. On Apple Silicon there is one memory, so “offloading” a layer does not copy it anywhere — it decides which processor executes that layer’s operations. Partial offload therefore costs less than it does on a PC, and also helps less: the layers left on the CPU do not save you GPU memory in the way a PC-shaped intuition expects.

Two escape hatches exist and they are not the same. --n-gpu-layers 0 keeps the backend loaded and moves the layers, and llama.cpp’s build documentation notes that the GPU may still accelerate parts of the computation at -ngl 0. To take Metal out of the picture entirely, the documented route is --device none. Use the second when you want a clean CPU baseline; use the first when you are bisecting a memory problem.

Thread count interacts with this more than people expect. With every layer on the GPU the CPU is doing sampling and orchestration, and a large -t buys nothing while adding contention; with layers split across both, the CPU side becomes the critical path and the thread count starts to matter again. Apple’s asymmetric cores make the naive default wrong in one specific way — counting efficiency cores into the thread pool slows the pool to their pace — which is the subject of the threads flag page. Set -ngl first, then tune threads against whatever is left on the CPU, and measure each change with the same llama-bench invocation.

The working-set ceiling

The number llama.cpp prints as recommendedMaxWorkingSetSize comes from Metal itself, and it is smaller than your installed RAM. It is the amount the system is willing to have resident for the GPU, and llama.cpp asks for allocations against it: weights, the KV cache and the compute buffer all come out of the same figure.

When a request exceeds it the failure is explicit rather than a slowdown — the Metal buffer allocation logs error: failed to allocate buffer, size = ... MiB and names the size it wanted. Compare that size against the ceiling in the init block and you have the diagnosis without guessing. The three levers, in the order that costs least: shorten the context, quantise the KV cache with -ctk and -ctv, then move to a smaller quant. Adding more layers to the CPU with a lower -ngl also works and is the bluntest of them.