What Fits on an RTX 4090's 24GB, by the Numbers
10 min read · updated August 11, 2026
Twenty-four gigabytes is not a model size. It is a budget that three things draw on, and the one people leave out of the sum is the one that grows with every token in the conversation.
What 24GB actually is
NVIDIA publishes the GeForce RTX 4090 as 24 GB of GDDR6X on a 384-bit interface, with a total graphics power of 450 W and a recommended system supply of 850 W, on its RTX 4090 product page (retrieved 11 August 2026). Two of those numbers matter here and the third matters on the power supply page.
The 24 GB is a marketing gigabyte — 24 × 109 bytes, which is 22.35 GiB. Every allocator you will meet works in binary units, so the first thing to do is convert once and then stay in GiB. From that 22.35 GiB, subtract the CUDA context, the runtime’s compute buffers and, on a machine with a monitor plugged in, the desktop’s own framebuffer. On a headless Linux box that overhead is small; with a browser open on a 4K display it is not. Budget 0.6–1.0 GiB for the runtime and read the desktop’s share off nvidia-smi rather than guessing — and see running the card headless if you want that overhead to be zero.
That leaves roughly 21.4–21.8 GiB of usable space. Everything below assumes 21.5 GiB and says so wherever it matters.
Three things routinely eat into that figure without appearing in anybody’s arithmetic. A second model kept resident — an embedding model for retrieval, a speech model, a small draft model for speculative decoding — is a full allocation of its own, not a rounding error. A server that holds separate KV caches for concurrent requests multiplies the cache term by the number of sequences it will hold, not by one. And allocation is not perfectly compact: a runtime that has loaded and unloaded models in one process can fail to find a contiguous block for a tensor while nvidia-smi shows gigabytes free, which reads as a phantom out-of-memory and is fixed by restarting rather than by shrinking anything.
Weights: bits per weight, from file sizes
“Q4” is not a number. It is a family of formats whose effective bits per weight differ by half a bit or more, because k-quant formats store some tensors at a higher precision than others. The only honest way to get bits per weight is to divide a published file size by the parameter count.
Take the GGUF repackages published on Hugging Face by bartowski, whose model cards list a size for every quant. For Meta-Llama-3.1-8B-Instruct-GGUF the card gives Q4_K_M at 4.92 GB and Q8_0 at 8.54 GB; for Llama-3.3-70B-Instruct-GGUF it gives Q4_K_M at 42.5 GB and IQ3_XXS at 27.5 GB (both retrieved 11 August 2026). Against 8.03e9 and 70.6e9 parameters respectively:
bits_per_weight = file_bytes * 8 / parameters 8B Q4_K_M : 4.92e9 * 8 / 8.03e9 = 4.90 bits/weight 70B Q4_K_M : 42.5e9 * 8 / 70.6e9 = 4.82 bits/weight 8B Q8_0 : 8.54e9 * 8 / 8.03e9 = 8.51 bits/weight
So Q4_K_M is not four bits, it is a little under five, and the gap is worth 0.9 GB on an 8B model and 8 GB on a 70B. That is the difference between a model fitting and not fitting, which is why the estimate “params ÷ 2 for Q4” is the single most common reason somebody’s load fails. See choosing a quantization for what the quality cost of each step down actually is.
At 4.82 bits/weight, a 70B does not fit 24 GB and no arrangement of the budget changes that: 39.6 GiB of weights against 21.5 GiB of space. IQ3_XXS at 27.5 GB is 25.6 GiB, still short. A 70B on one 24 GB card means partial offload to system RAM, which is a different page — see the 12GB page, where that cliff is the whole subject.
KV cache: the term everyone forgets
The attention cache holds one key and one value vector per layer per token, and its size is fixed by the model’s architecture, not by the quantization of the weights. From a model’s config.json:
kv_bytes_per_token = 2 * n_layers * n_kv_heads * head_dim * bytes_per_element Llama 3.1 8B (32 layers, 8 KV heads, head_dim 128, fp16) = 2 * 32 * 8 * 128 * 2 = 131,072 B = 128 KiB per token 8,192 tokens -> 1.0 GiB 32,768 tokens -> 4.0 GiB 131,072 tokens -> 16.0 GiB
Those layer and head counts are the published values in the model config — 32 hidden layers, 8 key-value heads, head dimension 128, maximum position embeddings 131,072 — visible in any mirror of the Llama 3.1 8B config.json. Grouped-query attention is doing enormous work here: with 32 query heads but only 8 KV heads, the cache is a quarter the size it would be under full multi-head attention. On a model without GQA the numbers below would look very different.
Two levers reduce this. Quantizing the cache to 8 bits halves it — llama.cpp exposes --cache-type-k and --cache-type-v, both accepting q8_0 — and enabling flash attention with -fa removes the large intermediate attention matrix, which is not the cache but is allocated alongside it.
What fits, derived
Now the budget closes. Usable 21.5 GiB, minus weights, leaves space for cache; divide by the per-token figure to get the context ceiling. All three lines below are arithmetic on the file sizes and config values already cited, with an fp16 cache assumed:
Llama 3.1 8B Q4_K_M weights 4.58 GiB -> 16.9 GiB free -> 135k tokens Llama 3.1 8B Q8_0 weights 7.95 GiB -> 13.5 GiB free -> 108k tokens Llama 3.2 3B Q4_K_M weights 1.88 GiB -> 19.6 GiB free -> 179k tokens
The first line contains the finding worth taking away. An 8B at Q4_K_M on a 24 GB card is not limited by VRAM at all — it is limited by the model, whose max_position_embeddings is 131,072. You can fill the entire documented context window and still have room left over. That inverts the usual advice: on this card, at this size, the useful question is not “how much context can I afford” but “what is the largest model I can run at the context I actually need”.
Run the same sum for a 32B-class model at Q4_K_M — roughly 19 GiB of weights — and the answer flips completely: two or three gigabytes of headroom, a few tens of thousands of tokens, and no room for a second model to stay resident. That is the regime where cache quantization stops being an optimisation and becomes the thing that makes the configuration possible.
Measuring it on your own card
None of the above is a speed claim, and it should not be read as one. The 4090’s 384-bit interface at the 21 Gbit/s per-pin rate its GDDR6X ships at gives 384 ÷ 8 × 21 = 1,008 GB/s, which sets a hard ceiling of about 205 tokens per second on a 4.92 GB model at batch size one, since every token must read every weight once. Real rates are below that ceiling by a margin that depends on the runtime and the build. To find yours, use llama.cpp’s own benchmark tool, documented in the llama-bench README:
# generation rate at three context depths, 5 repetitions each llama-bench -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \ -ngl 99 -p 512 -n 128 -r 5 -o md # watch the allocation while it loads, in a second shell nvidia-smi --query-gpu=memory.used,memory.total \ --format=csv --loop-ms=500