Fitting a Model and Its Context Window in 12GB of VRAM

Twelve gigabytes is exactly enough to make the wrong choice look reasonable. A 13B at Q4_K_M loads and reports no error, and then runs out of context in the middle of the first real document.

The budget

The equation is the same one used on the 8 GiB page: available context is what remains after the weights and the runtime, divided by the per-token cache cost.

C = (V - W - R) / k        V = 12 GiB, R assumed 1.0 GiB

so 11.0 GiB is available for weights plus cache.

The 1.0 GiB reserve is an assumption for a card that is not also driving a desktop. Measure it — the method is on the runtime overhead page— because on a 12 GiB card an extra 700 MiB of display framebuffer is the difference between two of the rows below.

Twelve gigabytes is a particularly unforgiving budget because of where it sits. On 8 GiB the choice is made for you: nothing above an 8B is plausible and you spend your time tuning context. On 24 GiB there is enough slack that most reasonable configurations work. Twelve is the size where two very different setups both load without complaint and only one of them is any good, and the runtime gives you no signal about which you have picked. The rest of this page is that comparison, done in numbers.

The 13B route

Llama 2 13B is 13,015,864,320 parameters, 40 layers, and — this is the decisive part — 40 key/value heads, meaning no grouped-query attention and 800 KiB of cache per token. Bits per weight throughout this page are the measured figures published in llama.cpp’s quantize README:

k = 2 * 40 * 40 * 128 * 2 = 819,200 bytes/token = 800 KiB

quant     weights     free      context
Q4_K_S      7.07 GiB   3.93 GiB   5,152 tokens
Q4_K_M      7.42       3.58       4,697
Q5_K_M      8.64       2.36       3,090
Q6_K        9.95       1.05       1,382
Q8_0       12.88       negative   will not load

with q8_0 cache (400 KiB/token):
Q4_K_M      7.42       3.58       9,394 tokens

Under five thousand tokens at the default quant and cache precision. That is roughly 3,500 words of English — a long email thread, not a codebase or a contract.

Notice how quickly the rows collapse. The gap between Q4_K_M and Q6_K is 2.53 GiB of weights, which is 21% of the card — but it is 71% of the free space, so the context falls by 71% too. At Q8_0 the weights alone exceed the card. The 13B occupies so much of a 12 GiB budget that the remainder behaves like a much smaller card, and every decision made in that remainder is amplified.

The 8B-at-a-higher-quant route

The same 12 GiB spent on Llama 3.1 8B (8,030,261,248 parameters, 32 layers, 8 KV heads, 128 KiB per token at fp16):

k = 2 * 32 * 8 * 128 * 2 = 131,072 bytes/token = 128 KiB

quant     weights     free      context
Q4_K_M      4.58 GiB   6.42 GiB  52,629 tokens
Q6_K        6.14       4.86      39,848
Q8_0        7.95       3.05      25,010

Mistral 7B at Q8_0: 7.17 GiB weights, 3.83 free, 31,403 tokens

A 12 GiB card runs an 8B at Q8_0 — near-lossless eight-bit weights — with 25,000 tokens of context. Or at Q6_K with 40,000. Both leave the weights far closer to the original model than any Q4 does.

The free space is doing something qualitatively different here. On the 13B, moving from Q4_K_M to Q5_K_M cost a third of the available context; on the 8B, moving from Q4_K_M all the way to Q8_0 — nearly doubling the bits per weight — still leaves 25,000 tokens. That is the whole benefit of a model that does not consume most of the card: decisions become cheap, and you can afford the precision and the window at the same time rather than trading one against the other.

Reading the two together

option                        weights     context     bits/weight
Llama 2 13B  Q4_K_M          7.42 GiB     4,697        4.89
Llama 3.1 8B Q8_0            7.95 GiB    25,010        8.50
Llama 3.1 8B Q6_K            6.14 GiB    39,848        6.56

For nearly identical memory, one option gives 60% more parameters at 4.89 bits each and 4,700 tokens; the other gives fewer parameters at 8.5 bits each and 25,000 tokens. The second is 5.3x the context and 1.7x the precision per weight. It is not a close call for most work, and it is invisible in any table that lists weights and stops.

The reason it is invisible is that the two disadvantages of the 13B route come from different places and neither is in the name. The context deficit is architectural — 40 key/value heads rather than 8 — and belongs to that generation of models rather than to that size. The precision deficit is budgetary: a larger model on a fixed card has to be quantized harder, so choosing more parameters is automatically choosing fewer bits for each of them. On a fixed budget those two choices are not independent, which is what makes “bigger model equals better” unreliable exactly where people most want to apply it.

There is a second reason the comparison favours the smaller model, and it is about speed rather than memory. Single-stream decode reads every resident weight once per token, so time per token tracks the weight bytes: 7.95 GiB against 7.42 GiB is roughly the same, but the 13B is also reading and writing a cache six times larger per token as the conversation grows. The argument is developed in why local inference is bandwidth-bound.

Where the 13B still wins

• Short, hard, single-turn tasks. If the whole job fits in 4,000 tokens and quality per token matters more than quantity, the extra parameters are real. Classification, extraction from a short document, a difficult single-shot rewrite.
• A fine-tune that only exists at 13B. Availability beats architecture. If the specific adapter or merge you need is a 13B, the arithmetic above tells you what context you get, not whether to use it.
• With a q8_0 cache and a short window on purpose. 9,394 tokens at Q4_K_M with a quantized cache is a workable assistant, provided nothing ever pastes a long file into it.

There is one more configuration that people reach for on this card and should not: a 34B at a low quant. A 34B needs 18.7 GiB at Q4_K_S, which is over a 12 GiB card by half again, so the only way it loads is a partial offload to system RAM — and offloading a third of the weights to a bus ten times slower costs a factor of several in tokens per second, derived on the overflow page. It will run. It will produce a few tokens a second, and the model you are running at that speed is one that has also been quantized hard. Neither half of that trade is good.

What does not work is running a 13B with the runtime’s default context and hoping. A modern model advertises a large window, the loader will try to allocate the cache for all of it, and on a 12 GiB card that fails at load — or worse, silently spills, which is a different and slower failure. Set -c explicitly, always.