Choosing a Quantization Level for Your Mac's Memory Budget

“Use Q4” is a rule of thumb that happens to be right on a lot of machines and silently wrong on the rest. The choice is a constraint with three inputs, two of which your Mac will tell you if you ask, and solving it takes a minute.

The equation

budget  >=  weights + cache + headroom

weights   = parameters x bits_per_weight / 8
cache     = tokens x 2 x layers x kv_heads x head_dim x kv_bytes
headroom  = activations and everything else you keep running

Every term is knowable in advance. Rearranged for the unknown you are actually choosing:

bits_per_weight  <=  8 x (budget - cache - headroom) / parameters

One correction before you substitute anything in: bits per weight is not the number in the model’s name. MLX’s default affine scheme stores the quantized value plus an fp16 scale and an fp16 bias for every group of 64 weights, so the real cost is:

nominal   real (group 64)   bytes per parameter
  2 bits    2.5 bits            0.3125
  3 bits    3.5 bits            0.4375
  4 bits    4.5 bits            0.5625
  6 bits    6.5 bits            0.8125
  8 bits    8.5 bits            1.0625
 fp16      16 bits              2.0

That is a twelve per cent surcharge at 4 bits and a twenty-five per cent surcharge at 2, which is exactly the range where people are counting gigabytes. It is confirmed against real conversions: a 70,553,706,496-parameter Llama-3.3-70B converts to 39,688,567,605 bytes at nominal 4 bits, which is 4.50 bits per weight. The derivation is in quantizing a model for MLX.

Input one: your actual budget

Not your installed RAM. macOS reports what the GPU may hold, and MLX surfaces it:

python - <<'PY'
import mlx.core as mx
d = mx.device_info()
gb = lambda n: round(n / 1e9, 2)
print("installed RAM   ", gb(d["memory_size"]), "GB")
print("max working set ", gb(d["max_recommended_working_set_size"]), "GB")
PY

max_recommended_working_set_size is Metal’s own answer and is always below installed RAM. Subtract from it whatever you actually keep open — a browser with a real session is commonly several gigabytes — and what remains is the budget. On a machine that is also your desktop, leaving four to six gigabytes for everything else is not conservative, it is the difference between a working machine and a swap-thrashing one.

Apple documents raising the underlying limit with sudo sysctl iogpu.wired_limit_mb, which enlarges the budget by taking from the operating system. Treat it as a last step rather than an input; the allocator controls that accompany it are in how MLX manages memory.

Input two: the context you need

The cache term is where budgets are usually blown, because it is invisible until the conversation is long. Every value in it comes from the model’s config.json:

cache bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_value

Llama-3.3-70B  80 layers,  8 kv_heads, 128 head_dim, fp16 -> 0.328 MB/token
Qwen3-14B      40 layers,  8 kv_heads, 128 head_dim, fp16 -> 0.164 MB/token

Multiply by the context you will genuinely use, not the maximum the model advertises. Eight thousand tokens on the 70B is 2.7 GB; thirty-two thousand is 10.7 GB; the architecture’s stated maximum of 131,072 is 43.0 GB, which is more than the 4-bit weights.

kv_bits divides this term directly — 4-bit cache is a quarter the bytes — and unlike weight quantization it costs very little at moderate context. If the equation does not balance, this is usually the cheapest term to attack first. The general behaviour is in the KV cache.

Solving for bits

Worked, for a machine reporting roughly a 48 GB working set, wanting 16k of context on a 70,553,706,496-parameter model, keeping 5 GB for the desktop:

cache at 16,384 tokens, fp16   = 16,384 x 327,680      =  5.37 GB
headroom                                                 =  5.00 GB
available for weights          = 48.0 - 5.37 - 5.00      = 37.6 GB

bits_per_weight <= 8 x 37.63e9 / 70,553,706,496          = 4.27 bits

-> nominal 4-bit costs 4.5. It does not fit.
   Options: --kv-bits 4 cuts the cache to 1.34 GB -> 41.66 GB for weights
            -> 4.72 bits available, which admits nominal 4-bit;
            or nominal 3-bit at 3.5 bits, which fits with room to spare;
            or a smaller model.

That is the whole method, and its value is that it produces a specific answer with a specific reason. The same machine wanting only 2k of context at fp16 cache has 42.33 GB for weights, which is 4.80 bits and admits nominal 4-bit outright — same hardware, same model, different answer, because the context requirement changed.

Run it once for the model you want and once for the model one size down. Frequently the second comes out at 6 or 8 bits within the same budget, and then the real question is whether a 14B at 8 bits beats a 70B at 3 on your task — a question about your work, not about the machine.

There is a second constraint the equation does not carry, and it bites from the other end. Bits per weight is also bytes read per token, so every bit you spend on precision is a proportional reduction in the decode ceiling: at a fixed bandwidth, nominal 8-bit reads 1.0625 bytes per parameter against 4-bit’s 0.5625, which is 1.89 times the traffic and therefore a ceiling 47 per cent lower. Fitting is binary and speed is continuous, so the right reading is that the highest quantization level that fits is not automatically the one to choose — it is the slowest one that fits.

Where arithmetic stops

The equation tells you what fits. It does not tell you what is good enough, and nothing in this cluster measured that.

What can be said from mechanism is where the loss concentrates. Quantization error is not uniform: transformer activations contain a few features with magnitudes far outside the rest of the distribution, and a group containing one has its scale stretched to cover it, spending most of the representable range on a single value. Smaller groups limit the blast radius; mixed-bit recipes protect the layers where it hurts most; below four bits the headroom that made either workable is gone. So degradation shows up first in long multi-step reasoning, exact recall of rare facts and strict adherence to a schema — and last in fluent short prose, which is why a quick chat with an aggressively quantized model feels fine and a week of real work with it does not.

Two practical consequences. First, evaluate on your hardest task rather than your typical one, because the typical one will not show you the difference. Second, when the equation says the model you want does not fit at a level you trust, that is a real answer — the honest options are a smaller model, a shorter context, or a hosted model for the requests that need the capability, not a quantization level you have talked yourself into.