How MLX Manages Memory on Apple Silicon

On a discrete GPU, a tensor exists in host memory or in device memory and something copies it across a bus. On Apple silicon there is one pool of physical memory and both processors address it, so MLX has no transfer step to optimise. What it has instead is an allocator, a cache, and a limit on how much of the shared pool the GPU is permitted to hold.

There is no copy to eliminate

Apple’s MLX documentation states the model directly: arrays live in unified memory, and any device can perform any operation on them without needing to move them from one memory location to another. You do not place an array; you place an operation, with stream=mx.cpu or stream=mx.gpu. Where two streams touch the same array, MLX inserts the dependency so the second does not start before the first finishes.

The consequence for inference is smaller than the marketing suggests but real. Weights are memory-mapped from the safetensors file and used in place, so loading a 39 GB model does not build a 39 GB staging copy first. A tokenizer step that runs on the CPU hands its output to a matmul on the GPU with no marshalling. And a KV cache that grows during generation grows in the same pool the weights occupy, which is why the two must be budgeted together rather than separately.

It also means the machine has no separate memory to run out of. A discrete card fails with a clean out-of-memory error and leaves the system running; a Mac that over-commits pushes the pressure into the operating system’s compressor and then into swap, and the symptom is a machine that becomes slow rather than a program that stops. That failure has its own page.

The buffer cache, and the leak that isn’t

MLX keeps freed Metal buffers rather than returning them to the system, because allocating a Metal buffer is expensive and a generation loop allocates and frees the same shapes thousands of times. The effect on Activity Monitor is that the process’s memory rises during a run and does not fall afterwards, which looks exactly like a leak and is not one.

Four functions in mlx.core control and inspect it:

• mx.set_cache_limit(n) — sets the free-cache limit in bytes. The documented behaviour is that if more than the limit is in use, memory is reclaimed from the cache on the next allocation. Setting it to 0 disables the cache. It returns the previous limit.
• mx.clear_cache() — releases the cached buffers now, without changing the policy. This is the one to call between loading one model and loading another.
• mx.set_memory_limit(n) — a ceiling on active memory rather than on the cache.
• mx.get_active_memory(), mx.get_peak_memory() and mx.reset_peak_memory() — what is live now, the high-water mark, and a way to zero the mark between phases so you can attribute a peak to prefill or to decode.

The wired limit

Unified memory does not mean the GPU may use all of it. macOS keeps a limit on how much memory can be wired — held resident and never paged out — and that number, not your total RAM, is the practical ceiling on a resident model.

MLX exposes it as mx.set_wired_limit(limit). Apple’s documentation for that function records four things worth quoting: it only functions on macOS 15.0 or higher, the value must be strictly less than the total memory size, the default is 0, and setting a wired limit above the system’s own limit raises an error. It returns the previous limit in bytes.

To raise the system limit itself, the same documentation gives the command:

sudo sysctl iogpu.wired_limit_mb=<size_in_megabytes>

Reading your machine’s real ceiling

Rather than assuming a percentage, ask the machine. mx.device_info() returns a dictionary whose keys, in the current Metal backend, are device_name, architecture, max_buffer_length, max_recommended_working_set_size, memory_size and resource_limit.

python - <<'PY'
import mlx.core as mx
d = mx.device_info()
gb = lambda n: round(n / 1e9, 2)
print(d["device_name"], d["architecture"])
print("total memory          ", gb(d["memory_size"]), "GB")
print("max working set       ", gb(d["max_recommended_working_set_size"]), "GB")
print("max single buffer     ", gb(d["max_buffer_length"]), "GB")
PY

memory_size is read straight from sysctl hw.memsize, so it is your installed RAM. max_recommended_working_set_size is Metal’s own answer to “how much may the GPU hold”, and it is the number to plan against; max_buffer_length is the largest single allocation, which is what a very large individual tensor runs into first.

Those two are also the numbers Apple’s documentation points you at when deciding what to pass to set_wired_limit: the system wired limit and the total memory. Two commands and you have the machine’s real budget instead of a rule of thumb, which is what the quantization budget page builds on.

Which lever to pull

They are not interchangeable, and reaching for the wrong one is common.

• The model does not fit at all. No allocator setting fixes this. Use fewer bits per weight or a smaller model.
• It fits, then fails part way through a long prompt. That is the KV cache growing. Cap it with max_kv_size or shrink it with kv_bits; the allocator is not the problem.
• It fits alone but not with everything else you have open. Cache limit and wired limit are the right knobs here, in that order —mx.clear_cache() costs nothing but time, and raising the wired limit trades the operating system’s headroom for yours.
• Memory climbs across many requests in a long-lived process. Check whether you are holding prompt caches. A per-conversation cache you never drop is real memory, and it is not the buffer cache.