Apple Silicon for Local Inference: Capacity Against Bandwidth

The interesting thing about a unified-memory machine is not that it is fast or slow. It is that it changes the two terms of the decode bound in opposite directions, so the comparison has an actual answer that depends on which term binds for you.

What unified memory actually changes

In a conventional desktop, the GPU has its own memory on its own board behind a PCIe link, and anything the GPU needs must be copied there first. A unified-memory system-on-chip puts CPU and GPU on one die sharing one physical memory pool with one address space. Two consequences follow, and both matter for inference:

• The accelerator’s memory is the machine’s memory. A configuration with a large RAM option makes all of it addressable by the GPU, so the capacity available to a model is not limited to what fits on a graphics board.
• There is no host-to-device copy. The transfer that makes offloading so painful on a discrete card does not exist here, because there is nowhere to transfer to.

The capacity argument

Put it through the same formula. At 4-bit weights and roughly 0.55 bytes per parameter in practice, the model size a machine can hold is about:

P_max ~= (usable_memory - KV - overhead) / 0.55

usable_memory on a unified system is total RAM minus what the
OS and applications need, and minus any cap the platform places
on how much a GPU process may allocate.

A machine configured with 64 GB, leaving perhaps 48 GB usable, reaches roughly 85B parameters of 4-bit weights before KV. A 128 GB configuration reaches into the range where a 100B-plus model or a large mixture-of-experts checkpoint is resident. There is no discrete consumer graphics card that holds those, at any price, because consumer VRAM is soldered in tiers that top out far below system RAM options.

That is the whole capacity case and it is genuinely strong: for large sparse models in particular, where total parameters must be resident while only the active ones are read per token, a unified machine can run a model an entire class above what its bandwidth would suggest.

One qualification belongs here rather than in a footnote. Platforms typically cap how much of the shared pool a single process may allocate for accelerator work, and the cap is a fraction of total RAM rather than all of it. Some platforms expose a setting to raise it; raising it too far starves the operating system and the machine begins swapping, which is a far worse outcome than a smaller model. Treat “usable memory” in the formula as the allocation ceiling minus a working margin, not as the number printed on the box.

The bandwidth counter-argument

Decode speed does not care how the memory is attached, only how fast it can be read. Unified memory on a system-on-chip is typically LPDDR across a wide bus; high-bandwidth memory on an accelerator is stacked DRAM with a far wider interface. The published bandwidth figures for the two categories have historically differed by roughly an order of magnitude at the top end, and the decode bound turns that difference into token rate one-for-one:

Example inputs — substitute your machine's datasheet figure.

70B model, 4-bit:  P * b = 70e9 * 0.55 = 38.5 GB per token

at   400 GB/s   ->  400 / 38.5   ~= 10 tok/s
at   800 GB/s   ->  800 / 38.5   ~= 21 tok/s
at  3000 GB/s   ->  3000 / 38.5  ~= 78 tok/s

Same model, same arithmetic, three answers, and the only variable is bandwidth. This is also why the bandwidth tier within a unified-memory product line matters far more than the core count: higher-tier chips widen the memory interface, and that widening is the number that shows up in generation speed.

It also produces a counter-intuitive buying rule. Within a product line, a configuration with more memory but the same interface width runs a larger model at the same tokens per second, while a configuration with a wider interface runs the same model faster. Those are different purchases and the specification sheet presents them as adjacent options on one axis. Decide which of the two terms binds for you before reading the options: if the model you want does not currently fit, buy capacity, and if it fits and generates too slowly, buy interface width.

The batching consequence is worth stating too, because it is the one place where the comparison is not close. Below the ridge point, extra concurrent sequences are nearly free on any device — but reaching the ridge requires the arithmetic throughput to have somewhere to go, and a unified machine’s ridge point sits at a much lower batch than a datacentre accelerator’s. For one user this is irrelevant. For serving several at once, the machine saturates early, and the per-user speed starts degrading at a concurrency where a server part would still be in its free region.

Prefill is the weaker half

Decode is memory-bound, so a unified machine competes on the term it is weakest in but by a bounded factor. Prefill is compute-bound, and here the gap is the arithmetic throughput gap, which is larger. The practical symptom is characteristic: generation feels reasonable, and then you paste a long document and wait noticeably before the first token appears.

Time to first token grows with prompt length divided by achievable FLOP/s, so long-context and document-heavy work is exactly where a unified machine is least comfortable, and short-prompt conversational work is where it is most comfortable. Prompt caching, where the runtime supports it, attacks precisely this term by reusing the keys and values for a repeated prefix.

The absence of a host-to-device copy deserves one more mention here, because it changes which failure mode you get. On a discrete card, exceeding VRAM means either an allocation failure or a fall back to offloading across a slow link, and the second is catastrophic for throughput. On a unified system there is no link to fall across; the model either fits within the allocation ceiling or the system begins swapping to storage, which is worse still but arrives more gradually. The practical advice differs accordingly: on a discrete card the thing to avoid is spilling, and on a unified machine the thing to avoid is sizing so close to the ceiling that ordinary system memory pressure pushes you over it.

Which workloads land on which side