Context Length or Quantization: Which Wins the Same VRAM Budget

You have a card with a fixed amount of memory and two things competing for it. Dropping from Q5 to Q4 frees some weight memory, and that memory can be spent on context instead. The exchange rate between the two is not a matter of opinion — it falls out of the parameter count and the attention geometry, and it is different for an 8B and a 70B.

What the budget is actually spent on

Three things occupy VRAM while a model is serving, and only two of them are usually thought about. The weights are the largest and the most obvious. The KV cache is the second, and it scales with how much context you have allocated rather than with how much you are using. The third is the compute buffer — scratch space for the largest batch of activations the runtime expects to process at once — which is small relative to the other two but is the reason a configuration that looks like it fits by a few hundred megabytes does not.

The question this page answers is the trade between the first two. If the total is fixed, every byte you stop spending on weights is a byte you can spend on context, and the useful form of the answer is “how many tokens per bit”.

The KV cache costs a fixed number of bytes per token

The cache stores one key vector and one value vector per attention layer per token. So the per-token cost is entirely decided by the model’s published architecture:

bytes_per_token = 2 (K and V)
                 x n_layers
                 x n_kv_heads
                 x head_dim
                 x bytes_per_element

Llama 3.1 8B, as published in Meta’s model card and its config.json, has 32 layers, 8 key/value heads under grouped query attention, and a head dimension of 128. At fp16 that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, or exactly 128 KiB per token. A 32,768 token context therefore reserves 4 GiB of KV cache before a single request arrives.

Llama 3.1 70B has 80 layers with the same 8 KV heads and the same head dimension: 2 x 80 x 8 x 128 x 2 = 327,680 bytes, or 320 KiB per token. Note what did not change. Grouped query attention pins the head count, so KV cost grew by the layer ratio — 80/32, or 2.5x — while the parameter count grew nearly ninefold. That divergence is the whole reason the exchange rate differs by model size. See how the KV cache works for what it is doing with those bytes.

What one bit per weight is worth

Weight memory is the parameter count times the average bits per weight, divided by eight. That makes the marginal value of a quantization tier easy to state: dropping the average by one bit frees n_params / 8 bytes.

For an 8.03B model that is 8.03e9 / 8 = 1.004e9 bytes, or about 0.93 GiB per bit. The llama.cpp quantize README’s own size table for Llama-3.1-8B is a useful check on that: it lists Q4_K_M at 4.58 GiB and 4.8944 bits per weight, and Q8_0 at 7.95 GiB and 8.5008 bits per weight. The gap is 3.37 GiB across 3.6064 bits, which is 0.934 GiB per bit — the arithmetic above, confirmed by the tool’s own measurements of file size. The table is in the llama.cpp repository, published by the ggml project.

The exchange rate, and why size changes it

Divide one by the other. On the 8B, one bit per weight frees 1.004e9 bytes, and each token of fp16 context costs 131,072 bytes, so the trade is 1.004e9 / 131,072 = about 7,660 tokens of context per bit dropped.

On the 70B, one bit frees 70.6e9 / 8 = 8.83e9 bytes, and a token costs 327,680 bytes: 8.83e9 / 327,680 = about 26,900 tokens per bit. The same decision buys three and a half times more context on the larger model, because weights scaled with the parameter count and KV scaled only with depth.

Reading that as a practical answer: on an 8B, going from Q8_0 to Q4_K_M frees roughly 3.4 GiB, which is roughly 27,000 extra tokens of fp16 context — a real but bounded gain, and one you pay for in quality across the whole model. On a 70B the same three-and-a-half bits is worth on the order of 95,000 tokens, which is often the difference between a model that can hold your document and one that cannot. The larger the model, the more the trade favours quantizing to buy context.

The direction reverses when the context you need is small. If you are running 4k contexts, the entire KV cache on an 8B is 512 MiB and there is nothing to win by quantizing harder — spend the memory on the higher tier and take the quality. The trade only becomes interesting once the context you want is a significant fraction of the weights.

The lever most people should pull first

Before dropping a quant tier, quantize the cache instead. llama.cpp exposes --cache-type-k and --cache-type-v, and setting both to q8_0 halves bytes_per_element from 2 to 1, which halves the entire per-token figure — 128 KiB becomes 64 KiB on the 8B. That doubles the context you can hold at a given quant tier without touching the weights at all.

llama-server -m model-Q5_K_M.gguf \
  -c 32768 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl all

The reason to reach for this first is that the two kinds of loss are not comparable. Weight quantization degrades the function everywhere, on every token, including the ones where the model was already marginal. KV quantization degrades stored attention state, and the V cache in particular tolerates it well because values are averaged rather than compared. Quantizing the K cache is the more aggressive half of the pair, and is the first thing to revert if long-context recall gets worse.

Whichever way you resolve it, work the arithmetic before you download anything. A 30 GB file that turns out to leave 2 GiB for KV is an hour of bandwidth spent on a configuration you cannot use, and the calculation above takes a minute.