How Much VRAM a 13B Model Needs at Each Quant Level
10 min read · updated August 11, 2026
A 13B at Q4_K_M is 7.42 GiB of weights, which sounds like a comfortable fit on a 12 GiB card. It is not, and the reason is a term most tables omit entirely: this generation of 13B uses multi-head attention, so its key/value cache costs 800 KiB per token.
13.02 billion parameters, derived
The canonical model at this size is Llama 2 13B, and the ones that followed it — the many fine-tunes and merges — keep its shape. From the config: 40 layers, hidden size 5120, 40 attention heads, 40 key/value heads, head dimension 128, feed-forward intermediate size 13,824, vocabulary 32,000.
per layer:
attention 4 * 5120 * 5120 = 104,857,600
ffn 3 * 5120 * 13824 = 212,336,640
norms 2 * 5120 = 10,240
-----------
317,204,480
x 40 layers = 12,688,179,200
+ embedding and output head 2*32000*5120 = 327,680,000
+ final norm 5,120
--------------
total 13,015,864,320 (13.02B)Note 4 * 5120 * 5120 for attention: with 40 attention heads and 40 key/value heads there is no grouping, so all four projections are square. That single fact is what this page is about.
It is also worth checking before you apply anything here to a model that merely calls itself a 13B. The size class has largely been replaced by models in the 12 to 15 billion range built on newer shapes, and those almost all use grouped-query attention: a modern 14B with 40 layers and 8 key/value heads costs 160 KiB per token rather than 800, a factor of five, and every conclusion below inverts for it. Open the model’s configuration and read four fields — layer count, hidden size, head count and key/value head count. If the last two are equal you are in the case this page describes; if the fourth is much smaller, use the arithmetic and not the conclusions.
Weights at each quant
Bits per weight comes from llama.cpp’s quantize README, which publishes measured values per type. Applying them to 13,015,864,320 parameters:
type bits/weight weights (GiB) Q2_K 3.1593 4.79 Q3_K_M 3.9960 6.05 Q4_K_S 4.6672 7.07 Q4_K_M 4.8944 7.42 Q5_K_M 5.7036 8.64 Q6_K 6.5633 9.95 Q8_0 8.5008 12.88 F16 16.0005 24.24 check: 13,015,864,320 * 4.8944 / 8 = 7.96e9 bytes = 7.42 GiB
800 KiB per token
Now the term that decides the outcome. With 40 layers, 40 key/value heads and head dimension 128:
per token, fp16: 2 * 40 * 40 * 128 * 2 = 819,200 bytes = 800 KiB 1,024 tokens 0.78 GiB 4,096 tokens 3.13 GiB 8,192 tokens 6.25 GiB 16,384 tokens 12.50 GiB 32,768 tokens 25.00 GiB compare Llama 3.1 8B, grouped-query, 8 kv heads: 128 KiB/token 32,768 tokens 4.00 GiB
The 13B is 1.6x the parameter count of the 8B and 6.25x its cache cost per token. Nothing about the marketing name predicts that. It is the product of two independent choices — more layers, and no grouping — and it is the reason a 13B feels so much heavier in practice than the weights table suggests.
The mechanism behind grouped-query attention explains why the saving is available at all. In multi-head attention each of the 40 heads computes its own queries, keys and values, so 40 key vectors and 40 value vectors have to be stored per token per layer. Grouped-query attention keeps all 40 query heads but has them share a smaller number of key/value heads — four or eight query heads reading the same stored key and value. The queries are recomputed every step and never stored, so nothing is lost from the cache side of the ledger; only the stored half shrinks. The published results for models trained this way put the quality cost close to nothing, which is why the technique was adopted across the board immediately after this generation and why a model of this shape is now an artefact of its release date rather than a design choice anybody would repeat.
One consequence is that the cost also lands on speed, not only on capacity. Attention has to read the whole cache on every generated token, so 800 KiB per token of stored context means that at 8,000 tokens the runtime is streaming 6.25 GiB of cache through the memory system for each token it produces — more traffic than reading the quantized weights. On a model with grouped-query attention the same window costs 1 GiB and the weights still dominate. A long session on this architecture therefore slows down as it grows, in a way that a grouped-query model does far less.
Where the cache overtakes the weights
Set the two terms equal and solve for context length. The weights at Q4_K_M are 7.42 GiB; the cache is 800 KiB per token:
C = 7.42 GiB / 800 KiB per token = 9,720 tokens Llama 3.1 8B at Q4_K_M, same calculation: C = 4.58 GiB / 128 KiB per token = 37,482 tokens
The 3.9x gap between those two crossovers is entirely architecture. The 13B has more weights to fill and a cache that fills far faster, and both effects push the crossover down. It is a useful number to carry because it converts an abstract worry into a threshold you can compare against a real workload: if the documents you paste are typically fifteen thousand tokens, this model spends most of its memory on them.
Past about ten thousand tokens, more than half the memory this model occupies is conversation rather than model. That is a useful thing to know before spending an evening hunting for a smaller quant: below the crossover, dropping from Q5_K_M to Q4_K_M buys you 1.22 GiB; above it, switching the cache to q8_0 buys you more than that and costs less quality. Both levers are described under how KV memory behaves across a long session.
What that means for a 12 GiB card
Assume 1.0 GiB for the runtime and its compute buffer — an assumption, replace it with a measurement. The context you have left on a 12 GiB card is:
available = 12.00 - weights - 1.00 Q4_K_M 12.00 - 7.42 - 1.00 = 3.58 GiB -> 3.58 GiB / 800 KiB = 4,697 tokens Q5_K_M 12.00 - 8.64 - 1.00 = 2.36 GiB -> 3,090 tokens Q6_K 12.00 - 9.95 - 1.00 = 1.05 GiB -> 1,382 tokens Q8_0 12.00 -12.88 - 1.00 = negative -> does not load with the KV cache in q8_0 rather than fp16, Q4_K_M gives 9,394 tokens
Under 5,000 tokens at the quant most people default to. That is a short document, not a long chat. Worse, it is under the default context of most chat front-ends, so the first thing that happens is either a failed allocation at load or a silent truncation of the system prompt a few turns in. Set the context explicitly with -c so the failure is at load time and visible; the three ways the ceiling presents itself are on the long-session page. The honest conclusion for a 12 GiB card is usually to run an 8B at Q6_K instead and keep 40,000 tokens of context, which is worked through in fitting a model and its context in 12 GiB. The comparison with the next size class up is in the 34B derivation, where grouped-query attention returns and the picture changes again.