How Much VRAM a 34B Model Needs at Each Quant Level

The 34B class is the awkward one. It is the largest thing that fits a single 24 GiB card, but only just, and only at one quantization level — and two of the models sold at this size have different enough shapes that a single table cannot describe both.

Two models, one number

Yi-34B and CodeLlama-34B are both “34B”. Their published configurations are not close. From 01-ai’s config.json for Yi-34B and the corresponding file for CodeLlama-34B:

                  layers  hidden  heads  kv_heads  ffn     vocab
Yi-34B                60    7168     56         8  20480    64000
CodeLlama-34B         48    8192     64         8  22016    32016

derived parameter counts:
Yi-34B          34,388,917,248   (34.39B)
CodeLlama-34B   33,744,232,448   (33.74B)

A 2% difference in parameters, from two completely different routes: Yi is deep and narrow, CodeLlama is shallow and wide. Both use grouped-query attention with eight key/value heads, so the class as a whole escapes the problem that makes a 13B so cache-hungry. But depth drives the cache too, and 60 layers against 48 is a 25% difference on that term.

The class exists because of the gap it fills. A 13B fits almost anything and a 70B fits almost nothing; 34B is the largest dense model that a single high-end consumer card can hold, and several labs targeted exactly that. That is also why the sizing is so tight — nobody was aiming for comfortable headroom, they were aiming at the card. The practical result is that this is the one size class where getting the runtime reserve wrong by half a gigabyte changes whether your configuration loads.

The two shapes also behave differently under quantization for a reason worth knowing. The bits-per-weight figures published by llama.cpp are measured on a model whose embedding and output tensors are 13% of its parameters; Yi’s 64,000-token vocabulary over a 7,168-wide hidden state puts those tensors at about 2.7% of the model, and CodeLlama’s 32,016-token vocabulary at about 1.6%. Since the quantization mixes treat those tensors specially, applying an 8B-derived bits-per-weight to a 34B is an approximation that runs slightly high. Expect the real files to land a little under the table below, which is the safe direction for a budget.

Weights at each quant

Parameter count times bits per weight, divided by eight. Bits per weight from llama.cpp’s quantize README:

type      bpw       Yi-34B    CodeLlama-34B
Q2_K      3.1593    12.65 GiB   12.41 GiB
Q3_K_M    3.9960    16.00       15.70
Q4_K_S    4.6672    18.68       18.33
Q4_K_M    4.8944    19.59       19.23
Q5_K_M    5.7036    22.83       22.41
Q6_K      6.5633    26.28       25.78
Q8_0      8.5008    34.03       33.39
F16      16.0005    64.06       62.86

worked: 34,388,917,248 * 4.8944 / 8 = 2.104e10 bytes = 19.59 GiB

Read the Q5_K_M row against a 24 GiB card and the shape of the problem is already visible: 22.83 GiB of weights on a 24 GiB card leaves 1.17 GiB for the runtime and the entire context. It does not work.

The context term

Yi-34B         2 * 60 * 8 * 128 * 2 = 245,760 B/token = 240 KiB
CodeLlama-34B  2 * 48 * 8 * 128 * 2 = 196,608 B/token = 192 KiB

fp16 cache at:
                Yi-34B    CodeLlama-34B
  4,096          0.94 GiB      0.75 GiB
  8,192          1.88          1.50
 16,384          3.75          3.00
 32,768          7.50          6.00

Both are far better behaved than a 13B, which costs 800 KiB per token — a 34B with grouped-query attention is cheaper per token of context than a 13B without it, despite being 2.6x the size. This is the clearest case in the whole size ladder that parameter count does not predict memory on its own.

Yi’s 25% penalty relative to CodeLlama is worth roughly 5,500 tokens on a 24 GiB card, which is the difference between a working document assistant and one that truncates. It comes entirely from depth: 60 layers each storing keys and values against 48. Depth and key/value head count are the only two architectural terms in the cache formula, so between two models of the same nominal size the deeper one is always the one with less room for context, regardless of which has more parameters. Yi-34B happens to have both more parameters and more layers; the two effects are independent and here they point the same way.

The 24 GiB boundary, exactly

A 24 GiB card — NVIDIA publishes 24 GB GDDR6X on a 384-bit interface for the GeForce RTX 4090 — with 1.2 GiB assumed for runtime and compute buffer. Solving for context:

context = (24.00 - weights - 1.20) / kv_per_token

Yi-34B         Q4_K_S  24.00-18.68-1.20 = 4.12 GiB -> 17,980 tokens
Yi-34B         Q4_K_M  24.00-19.59-1.20 = 3.21 GiB -> 14,006 tokens
Yi-34B         Q5_K_M  24.00-22.83-1.20 = negative -> will not load
CodeLlama-34B  Q4_K_M  24.00-19.23-1.20 = 3.57 GiB -> 19,513 tokens
CodeLlama-34B  Q5_K_M  24.00-22.41-1.20 = 0.39 GiB ->  2,153 tokens

So the answer to “does a 34B fit in 24 GiB” is: at Q4, with roughly 14,000 to 19,500 tokens of context, and not at Q5. The 1.2 GiB runtime assumption is doing real work in that conclusion — if your card also drives a display it is larger, and the CodeLlama Q5_K_M row goes to zero. Measure it rather than inheriting this number; the method is in the runtime overhead page.

There is a second, quieter constraint on this class that has nothing to do with VRAM: loading. A 19.6 GiB file has to be read from disk into the card, and the runtime memory-maps it, so the first load of a cold file is bounded by storage. On an NVMe drive that is seconds; on a spinning disk or a network share it is minutes, and it repeats every time the process restarts unless the file stays in the page cache. On a machine with 16 GB of system RAM the page cache cannot hold a 19.6 GiB file, so every restart re-reads all of it. That is not a memory error, it just makes the model feel broken on the first request after every restart.

What is left when it does not fit

• Quantize the KV cache. Moving the cache to q8_0 halves the per-token cost, taking Yi-34B at Q4_K_M from 14,006 to roughly 28,000 tokens. This is usually a better trade than dropping the weights another level, because the cache is a store of activations rather than of learned structure.
• Drop to Q4_K_S or an i-quant. Q4_K_S is 4.6672 bits/weight against Q4_K_M’s 4.8944, which is 0.91 GiB on a 34B — nearly 4,000 extra tokens of Yi context. Whether the quality cost is acceptable is a question about your task, discussed under choosing a quantization level.
• Offload some layers to system RAM. Available, and expensive: the derivation of how expensive is in what happens when a model does not fit.
• Go down a class rather than down a quant. A 13B at Q8_0 is 12.88 GiB and leaves room for 13,000 tokens on the same card; an 8B at F16 is 14.96 GiB and leaves 64,000. Whether either is better than a 34B at Q4 is a quality question, not a memory one.

The order of those four matters. Cache precision and quant level are both free to change and reversible in seconds, so try them first and measure whether the result is acceptable on your own task. Offloading is the one that quietly makes everything slow, and going down a size class is the one that changes what the model can do — so they are the last two to reach for, in that order, and only after the cheap levers have been spent. A configuration that needed all four is a configuration that wanted a different card.