What MLX Is, and Why It Exists Alongside llama.cpp

MLX is not an inference engine. It is an array framework with a NumPy shaped API, built by Apple, and the two decisions that make it interesting for local models are both visible in the first ten lines of any program you write with it.

What it actually is

The package you install is mlx, and it gives you mlx.core for arrays and operations, mlx.nn for layers and modules, and mlx.optimizers for training. If you have written NumPy or PyTorch, almost nothing about the surface will surprise you: mx.array, mx.matmul, mx.softmax, a Module base class with parameters, function transforms like mx.grad and mx.vmap and mx.compile.

Running a language model is a separate package built on top of it. mlx-lm holds the model implementations, the Hugging Face conversion tooling, the generation loop and a server; mlx-whisper holds speech recognition. Installing mlx-lm pulls mlx in as a dependency, which is why most people meet the framework without ever importing it directly.

The hard requirements are narrow and worth knowing before you spend an afternoon on it. Apple’s installation documentation states that MLX requires an Apple silicon M series machine, macOS 14.0 or newer, and a native Python of 3.10 or newer. The failure mode when the last of those is wrong is confusing enough that Apple documents the check explicitly: run python -c "import platform; print(platform.processor())" and expect arm. If it prints i386 you are on an x86 Python under Rosetta and nothing will import.

Apple’s MLX installation page carries both the requirement list and that troubleshooting step.

Nothing runs until you ask

The first design decision is that operations do not execute when you write them. Apple’s documentation puts it plainly: a compute graph is recorded, and the actual computation happens only when an evaluation is forced. Printing an array forces it. Converting to NumPy forces it. Saving it forces it. Calling mx.eval forces it explicitly.

import mlx.core as mx

a = mx.ones((4096, 4096))
b = a @ a          # nothing has been multiplied yet
c = b.sum()        # still nothing
mx.eval(c)         # now the whole graph runs, once

This is not a performance trick bolted on afterwards; it is what makes the function transforms possible, because mx.grad needs a graph to differentiate rather than a result. But for local inference the consequence that matters is a memory one, and Apple’s lazy evaluation page gives exactly this example: constructing a model normally initialises every weight as float32, and lazily it does not initialise anything at all. You can then replace the parameters with float16 weights loaded from disk, and the float32 copies are never materialised. On a machine where the model and the operating system share one pool, never allocating a temporary is worth more than freeing one quickly.

The tuning knob is where you put the evaluation. Apple’s guidance is that anything from a few tens to many thousands of operations per evaluation is fine; the failure is at the extremes, where a graph is either so small that the dispatch overhead dominates or so large that the intermediates it holds alive stop fitting.

Arrays that no device owns

The second decision follows from the hardware. On Apple silicon the CPU and GPU address the same physical memory, so MLX has no concept of moving an array to a device. Apple’s unified memory page states that arrays live in unified memory and that any device can perform any operation on them without needing to move them from one memory location to another.

What you specify instead is where an operation runs, through a stream argument:

c = mx.add(a, b, stream=mx.cpu)
d = mx.add(a, b, stream=mx.gpu)   # same a and b, no copy

There is no .to(device) and no .cuda(), because there is nowhere for the data to go. Where two streams touch the same array MLX inserts the dependency itself, so a GPU operation that consumes a CPU result will not start early. Apple’s own worked example on that page splits a workload across both processors and reports a roughly twofold speedup over the GPU alone on an M1 Max — the point being not the ratio but that mixing the two costs no transfers.

This is the piece that has practical consequences you can feel, and it is covered properly in how MLX manages memory on Apple Silicon, including the buffer cache and the wired-memory limit that decide how much of your RAM the GPU is allowed to touch.

Where llama.cpp sits

The two projects are not competing implementations of one idea; their core abstractions are different things.

• llama.cpp’s unit is a quantized model file. It is a C and C++ inference engine built around GGUF, with backends for Metal, CUDA, Vulkan, ROCm and plain CPU. Its reach is the point: one file format, many machines, no Python. Its Metal backend is one target among many and does not get to assume unified memory.
• MLX’s unit is an array. It is a general framework that happens to have language models written in it, in Python, in a few hundred readable lines each. It reads safetensors, not GGUF. It trains as well as it infers — LoRA fine-tuning is in mlx-lm rather than a separate project — and it assumes Apple silicon everywhere, which is why it can drop the device concept entirely.

The practical read: if you want one command that runs on a Mac laptop and a Linux box with an NVIDIA card, that is llama.cpp, and the llama.cpp guide covers it. If you are on a Mac and want to modify the model, quantize it yourself, fine-tune it, or call it from Python without a subprocess, that is MLX. Plenty of people have both installed and it is not a contradiction.

Where MLX stops

Being Apple-only is not a limitation you can work around; it is the premise. There is no CUDA path for the Metal kernels and no meaningful MLX story on an Intel Mac, so anything you build on it is pinned to Apple silicon, and a service that must also deploy to a Linux GPU host will end up with two runtimes.

Model coverage is narrower than llama.cpp’s, because each architecture is a hand-written Python module rather than a loader for a general file format. A brand-new architecture usually appears in llama.cpp first and in mlx-lm within days to weeks; a rare one may never appear. And because MLX does not read GGUF, the enormous existing library of GGUF quantizations is not directly usable — you convert from the original checkpoint instead, which is what converting a model to MLX format is about, or you pull a pre-converted one from the mlx-community org.

None of that changes the ceiling on how fast a model can decode, which is set by memory bandwidth and not by the framework. That argument is made in full in why Apple Silicon punches above its GPU compute.