How Many Parameters Fit in 16GB of Unified Memory at Q4

The arithmetic says 28 billion parameters. The answer is closer to 13 billion, and the entire difference is in one term that Apple sets and most calculators ignore.

The formula, inverted

Weights are P * bpw / 8 bytes. Solve for P:

P = B * 8 / bpw          B = memory budget in bytes

at Q4_K_M (4.8944 bits/weight, from llama.cpp's published table):

  B = 16.0 GiB  ->  P = 28.1e9   parameters
  B = 12.0 GiB  ->  P = 21.1e9
  B = 11.2 GiB  ->  P = 19.7e9
  B = 10.7 GiB  ->  P = 18.8e9

at Q4_K_S (4.6672):  16.0 GiB -> 29.4e9
at IQ4_XS (4.4597):  16.0 GiB -> 30.8e9

Twenty-eight billion parameters in sixteen gigabytes is the number the formula gives and it is not achievable, because the budget is never the installed memory.

The inversion is worth having as a habit even though its first answer is wrong, because it turns a shopping question into an arithmetic one. Once you have P = B * 8 / bpw you can answer “what would I need for this model” as readily as “what fits in what I have”, and the two directions use the same three numbers. It also makes the sensitivity obvious: parameter count is linear in the budget and inverse in the bits, so a 10% larger budget buys 10% more parameters, and dropping from 4.89 bits to 4.46 buys 10% more without spending anything.

16 GB is not 16 GB of budget

Unified memory is shared between the CPU and the GPU, which is the feature — no copy across a bus, and no separate VRAM figure to fit under. It also means the operating system is a tenant. Three deductions:

• macOS itself. The window server, the compositor and everything you have open are in the same pool. A machine that is also your desktop cannot give the model everything.
• The GPU working-set limit. Apple’s Metal API exposes recommendedMaxWorkingSetSize on MTLDevice, the amount of memory the device is advised to use, and Metal backends including llama.cpp’s read it and warn or fail when an allocation would exceed it. It is a fraction of installed memory chosen by the system, not the whole of it, and the fraction differs between machines and macOS versions.
• Wired-memory policy. The kernel parameter iogpu.wired_limit_mb governs how much can be wired for GPU use. It can be raised, at the cost of leaving macOS less to work with, and raising it to the full installed memory will make the machine swap rather than make the model fit.

The number to use is the one your machine reports, not a rule of thumb. llama.cpp prints the Metal device’s recommended working set size in its startup log, and sysctl iogpu.wired_limit_mb shows the current wired limit — zero means the system default is in force.

The failure mode when you get this wrong is different from a discrete card’s, and worse. A GPU that runs out of memory raises an allocation error and the process stops. A Mac that runs out of unified memory starts compressing and swapping instead, because that is what the operating system does with memory pressure — so rather than failing, the whole machine becomes slow, the model included, and nothing in the runtime’s log says why. If a local model on Apple silicon is producing a token every few seconds and the memory pressure indicator is not green, that is the diagnosis, and the fix is a smaller model rather than a tuning flag.

The realistic ceiling

Take a 16 GB machine that is also being used as a computer, and assume it will give the model something in the region of 10.7 to 11.2 GiB before it starts fighting the operating system. That is an assumption — read yours — and it produces:

budget 10.7 GiB at Q4_K_M -> 18.8B parameters
budget 11.2 GiB at Q4_K_M -> 19.7B parameters

against real models:
  Llama 3.2 3B    3.21e9 params   1.83 GiB   comfortable
  Mistral 7B      7.24e9          4.13 GiB   comfortable
  Llama 3.1 8B    8.03e9          4.58 GiB   comfortable
  Llama 2 13B    13.02e9          7.42 GiB   fits, little room for context
  Yi-34B         34.39e9         19.59 GiB   does not fit at any k-quant

So the honest answer to “how many parameters fit in 16 GB of unified memory at Q4” is: about 19 billion of pure weights, which in practice means the largest comfortable model is a 13B, and a 34B is out of reach without dropping to two bits per weight. A model in the 20-billion range would fit its weights and nothing else.

The gap between 28.1 and 19 is worth attributing rather than absorbing, because it is not one deduction. Roughly five gigabytes is the operating system and the applications you have open — recoverable, in the sense that a machine doing nothing else gets some of it back. The rest is the working-set policy, which is not recoverable without changing a kernel parameter and accepting the consequences. Anyone quoting a parameter ceiling for a given amount of unified memory is implicitly assuming both, and their assumption about the second one is probably not the same as your machine’s.

Then subtract the context

The ceiling above is weights only. Reserve for the cache and re-solve:

P = (B - kv) * 8 / bpw

Llama 3.1 8B shape (128 KiB/token, fp16), 8,192 tokens = 1.00 GiB:
  (10.7 - 1.0) GiB at Q4_K_M -> 17.0e9 parameters

Llama 2 13B shape (800 KiB/token), 8,192 tokens = 6.25 GiB:
  (10.7 - 6.25) GiB at Q4_K_M -> 7.8e9 parameters
  — which is less than 13 billion, so a 13B with 8k of context
    does not fit this budget at all

That last line is the practical result and it depends entirely on the attention configuration rather than on the parameter count. A 13B with multi-head attention needs 6.25 GiB of cache for an 8,192-token window; an 8B with grouped-query attention needs 1.00 GiB for the same window. On a 16 GB machine the second choice is not a compromise, it is the only one that leaves room to work. The derivation of the cache term is on the 13B page.

Which reframes the whole question. “How many parameters fit” is the wrong thing to optimise on a machine this size, because the answer depends on an architectural property that the parameter count does not mention. The useful version is: which model leaves the most room for the context you need, at the highest bits per weight you can afford. On 16 GB that is almost always an 8B-class model with grouped-query attention at Q5 or Q6, and the ceiling derived above is mainly useful for ruling things out.

The other ceiling

Fitting is not the only constraint on a unified-memory machine. Single-stream decode reads every weight once per token, so the upper bound on tokens per second is the memory bandwidth divided by the weight bytes — and unified memory bandwidth on the base configurations of a given chip generation is a fraction of what the higher-tier chips offer, on the same amount of installed memory.

tok/s ceiling = bandwidth / weight_bytes

a 13B at Q4_K_M is 7.42 GiB = 7.97e9 bytes
  at 100 GB/s ->  12.5 tok/s ceiling
  at 400 GB/s ->  50.2 tok/s ceiling

an 8B at Q4_K_M is 4.58 GiB = 4.92e9 bytes
  at 100 GB/s ->  20.3 tok/s ceiling
  at 400 GB/s ->  81.3 tok/s ceiling

Ceilings, not measurements — no runtime reaches its roofline, and Apple publishes the memory bandwidth for each chip, so substitute yours. The point is that on the same 16 GB, the model that fits with room to spare is also the one that runs at a usable rate, and the two constraints push the same way. The argument behind the roofline is in why local inference never saturates a GPU.