How Much VRAM a 70B Model Needs at Each Quant Level
10 min read · updated August 11, 2026
A 70B at Q4_K_M is 40.2 GiB of weights. That figure is derived below from the parameter shapes, and it can be checked: llama.cpp publishes 43.1 GB for exactly this model at exactly this quant, which is 40.1 GiB. The derivation and the published number agree to within a third of a percent.
70.55 billion, not 70
Llama 3.1 70B: 80 layers, hidden size 8192, 64 attention heads, 8 key/value heads, head dimension 128, feed-forward intermediate 28,672, vocabulary 128,256 with an untied output head.
per layer:
q 8192*8192 = 67,108,864
k 8192*1024, v 8192*1024 = 16,777,216
o 8192*8192 = 67,108,864
ffn 3 * 8192 * 28672 = 704,643,072
norms 2*8192 = 16,384
-------------
855,654,400
x 80 = 68,452,352,000
+ 2 * 128256 * 8192 = 2,101,346,304
+ final norm 8,192
--------------
70,553,706,496 (70.55B)The k and v projections are 8192×1024 rather than square, because eight key/value heads at head dimension 128 is 1024 columns. Grouped-query attention saves 100 million parameters per layer here, and far more in the cache.
The feed-forward blocks are 82% of the model. That is worth noticing because it is where the quantization mixes do most of their work: the down-projection tensor alone is 235 million parameters per layer, and the policy that upgrades half of those to a higher-precision type is what makes Q4_K_M cost 4.89 bits per weight instead of 4.5. On a model this size that policy is worth about three gigabytes, which is the difference between two cards and three at some quant levels. The reconstruction is on the FP16-to-Q4 page.
The table, and its check
type bpw weights Q2_K 3.1593 25.9 GiB Q3_K_M 3.9960 32.8 Q4_K_S 4.6672 38.3 Q4_K_M 4.8944 40.2 Q5_K_M 5.7036 46.8 Q6_K 6.5633 53.9 Q8_0 8.5008 69.8 F16 16.0005 131.4 check against a published figure: derived 70,553,706,496 * 4.8944 / 8 = 43.16e9 bytes = 40.2 GiB llama.cpp README, Llama 3.1 70B at Q4_K_M: 43.1 GB = 40.1 GiB
That second line is from the Memory/Disk Requirements table in llama.cpp’s quantize README, which also gives 4.9 GB for the 8B and 249.1 GB for the 405B at the same quant. Two independent numbers agreeing is the reason to trust the rest of this cluster: the same method produces the 34B and 13B tables, and it has been checked here against a source that measured the file.
The agreement is also informative about which direction the residual error runs. The derivation applies a bits-per-weight measured on the 8B to a model whose embedding and output tensors are a smaller fraction of the total, so it should overshoot slightly — and it does, by 0.1 GB. On a 405B the same method against the README’s published 249.1 GB implies about 4.91 bits per weight, which is the same small overshoot at ten times the scale. An error that stays proportional as the model grows is a systematic one you understand, which is worth more for budgeting than a smaller error you cannot explain.
Context
per token, fp16: 2 * 80 * 8 * 128 * 2 = 327,680 bytes = 320 KiB 4,096 tokens 1.25 GiB 8,192 tokens 2.50 GiB 16,384 tokens 5.00 GiB 32,768 tokens 10.00 GiB 131,072 tokens 40.00 GiB
At the model’s full 128k context the cache is the same size as the Q4_K_M weights. Grouped-query attention is doing a great deal of work: without it, at 64 key/value heads, the same table would read 2.56 MiB per token and 320 GiB at full context, which is why every model at this scale now uses it.
The cache is also the term that decides how much of a multi-card budget you can actually spend on weights. A configuration that leaves only two gigabytes free after the weights is a 6,500-token model, and 6,500 tokens is not what anyone runs a 70B for. If you are sizing a machine rather than fitting one you already own, work backwards: pick the context you need, multiply by 320 KiB, add that to the weights, add per-device runtime, and then choose the quant that fits what is left. At 32,768 tokens the cache is 10 GiB, which on two 24 GiB cards leaves about 35 GiB for weights — enough for Q3_K_M and not for Q4_K_M.
How many cards that is
Weights plus an 8,192-token fp16 cache, plus roughly 1.2 GiB of runtime per card — the runtime cost is per device, so it multiplies with the card count rather than being paid once:
quant weights + 8k cache 24 GiB cards 48 GiB cards Q2_K 28.4 GiB 2 1 Q3_K_M 35.3 2 1 Q4_K_S 40.8 2 1 Q4_K_M 42.7 2 1 Q5_K_M 49.3 3 2 Q6_K 56.4 3 2 Q8_0 72.3 4 2 F16 133.9 6 3
Two 24 GiB cards at Q4_K_M is 48 GiB of capacity against 42.7 GiB of demand, and the margin is thinner than it looks because a layer cannot be split across cards — the runtime assigns whole layers, so the split is quantized to 80 units and the two cards will not be exactly even. Budget for the larger half. The mechanics of splitting are covered under running one model across several GPUs.
Two details in that table are easy to get wrong. The first is that the runtime reserve is per device: two cards means two CUDA contexts, two compute buffers and, if either drives a display, two framebuffers. It does not amortise. The second is that the key/value cache is split along with the layers, so each card holds the cache for the layers it owns rather than a full copy — which is the reason the split works at all, and why a 32k window on a 70B is feasible across two cards when it would not be on one card of twice the size minus the overhead.
A third detail applies before anything reaches the GPU. Loading a 40 GiB file means reading 40 GiB, and if the runtime memory-maps the weights the operating system will happily keep them in the page cache, so a machine with 32 GB of system RAM re-reads from disk on every restart. Sizing system memory to at least the file size is the difference between a five-second reload and a five-minute one, and it is invisible in every VRAM table.
The single-card question
It does not fit. The smallest k-quant, Q2_K, is 25.9 GiB of weights alone, which is over a 24 GiB card before the runtime allocates anything. The i-quants go lower — llama.cpp’s README publishes IQ2_M at 2.9294 bits/weight, which on 70.55 billion parameters is 24.0 GiB, and IQ2_XXS at 2.3824, which is 19.6 GiB — so a 70B on one 24 GiB card means roughly two bits per weight and whatever quality that costs. At that point a 34B at Q5 occupies similar memory and is a more defensible choice; see the 34B derivation.
It is also the point where a sparse model becomes the better answer to the same question. A mixture-of-experts model with a total in this range reads only its active parameters per token, so it decodes far faster than a dense 70B while occupying similar memory — the trade is derived on the MoE memory page. If the goal is capability per gigabyte rather than a specific checkpoint, that is the direction the arithmetic points.
The other route is to keep some layers in system RAM, which works and is slow for a reason you can compute: the throughput cliff from spilling to system memory.