Why Apple Silicon Punches Above Its GPU Compute for Local LLMs

A Mac has a fraction of a datacentre accelerator’s arithmetic throughput and beats it comfortably at one specific job: generating tokens for one user from a model too large to fit on a consumer graphics card. That is not a paradox and it is not marketing. It follows from a ratio.

The claim

The claim is narrow, and stating it narrowly is what makes it defensible: for autoregressive decoding at batch size one, the hardware characteristic that predicts speed is memory bandwidth, and arithmetic throughput barely enters. Everything else about Apple silicon’s suitability — the unified pool, the power draw, the absence of a separate card — is downstream of that or is a different argument.

The corollary is more useful than the claim. If decode is bandwidth-bound then a chip with modest FLOPs and generous GB/s is well matched to it, and a chip with enormous FLOPs and the same GB/s is not better at it. The usual way of ranking accelerators, by arithmetic throughput, is measuring a quantity that is not the constraint.

Two operations per byte

Take the ratio directly. Generating one token means one forward pass over the model. Each weight is loaded once and participates in a multiply and an add. So the pass performs approximately two floating-point operations for every parameter, and moves one weight’s worth of bytes for every parameter.

At 4-bit MLX quantization a weight occupies 4.5 bits, or 0.5625 bytes. The arithmetic intensity of the operation is therefore:

2 FLOPs per parameter / 0.5625 bytes per parameter  =  3.6 FLOPs per byte

Now compare that to what the hardware wants. Any modern processor — Apple’s, NVIDIA’s, anyone’s — has a ratio of peak arithmetic to peak bandwidth in the hundreds of operations per byte. A workload offering 3.6 is two orders of magnitude below the balance point. The arithmetic units are idle almost the entire time, waiting for weights to arrive.

That is the whole mechanism. Decode is a streaming read of the model through some multiply-accumulate units that could have handled far more if only the data had turned up faster.

Every GPU is starved, and it shows

The consequence is that at batch size one, every accelerator is running at a small fraction of its rated arithmetic throughput, and the ones with the most FLOPs are running at the smallest fraction. Their advantage is real but it is banked elsewhere: in prefill, in training, and in serving many users at once.

Two structural facts then favour Apple silicon for the single-user case, and both are consequences of putting the memory on the package rather than on a board:

• The bandwidth is high relative to the arithmetic. The memory sits next to the compute on a wide interface, so the GB/s per unit of arithmetic throughput is unusually generous compared with a consumer graphics card built for rendering.
• The capacity is not partitioned. A consumer graphics card with 24 GB of video memory cannot run a 39.7 GB model at all, at any speed, and there is no configuration option that changes that. A Mac with 64 GB of unified memory can, because there is one pool and the model is allowed to occupy most of it. Being slow at a model is a different category of outcome from being unable to load it.

The second point does most of the practical work. The comparison people actually make is not “Mac against H100”; it is “Mac against the graphics card I could afford”, and there the deciding factor is usually capacity rather than throughput. The specific ceilings are derived in unified memory bandwidth and tokens per second.

Where the argument reverses

An argument this clean has an equally clean set of exceptions, and pretending otherwise is how this position becomes fan writing. There are three, and each is the same condition — arithmetic intensity rising — appearing in a different guise.

• Prefill. Processing a prompt puts every prompt token through the model at once, which turns matrix-vector products into matrix-matrix products and raises intensity by roughly the token count. Prefill is compute-bound, so the FLOPs ranking reasserts itself, and a workload of very long prompts and very short answers is one Apple silicon is not especially good at.
• Batching. Serving many sequences at once reads the weights once for all of them, so intensity rises with the batch size and the problem becomes compute-bound again. This is precisely why hosted inference is cheap and local inference is not: a provider amortises one pass over the weights across dozens of users, and you amortise it across one. That is the real economics behind local versus API cost.
• Training and fine-tuning. Backward passes are dense arithmetic with high reuse and are compute-bound throughout. A Mac can fine-tune with LoRA on modest data and will lose badly to a datacentre GPU on anything larger, and no amount of memory bandwidth changes it.

What actually follows

The practical conclusions are unglamorous, which is a sign the argument is load-bearing rather than decorative.

When choosing between Macs, compare the GB/s figure and the memory tier, not the GPU core count — unless your workload is prompt-heavy, in which case invert that. When someone quotes a tokens-per-second figure, ask what batch size and what context length, because both change the answer by more than the hardware does. When a model is too slow, the first question is whether you are near the bandwidth ceiling for its byte count — if you are, the fix is fewer bytes, meaning a smaller model or fewer bits, and no amount of tuning will help.

And when the answer is that a local model cannot do the job at a tolerable speed, that is a real answer rather than a failure of configuration. The ceiling is arithmetic.