llama.cpp's --ctx-size Flag and What It Costs in Memory

--ctx-size is the only flag in llama.cpp whose memory cost you can compute exactly before you run anything, from five numbers the loader prints. It is also the flag most often set by doubling until something breaks.

The default is 0, which does not mean zero

On current master -c/--ctx-size defaults to 0, and 0 is documented as “loaded from model” — the context length the model was trained with, which the loader reports as n_ctx_train. For a Llama 3.1 derivative that is 131072, and a full-length KV cache for it is very large, so the parameter fitter normally steps in and reduces the context to something that fits before the run starts.

That reduction has a floor of 4096 tokens. It is easy to read the 4096 as the default; it is not. It is the smallest context the fitter will settle for while trying to make an unset configuration fit, which is why a machine that is short on memory tends to end up at exactly 4096 and a machine with room ends up somewhere much higher. If you need a specific number, set it: an explicit -c is not fitted, and the run will fail loudly rather than quietly give you less.

The formula, from numbers llama.cpp prints

The KV cache stores one key vector and one value vector per token per layer. Its size is therefore linear in context — not quadratic, which is attention’s compute cost, not its storage cost. Per layer, the widths are printed at load as n_embd_k_gqa and n_embd_v_gqa, and the total is:

kv_bytes = n_ctx
         * sum over layers of (n_embd_k_gqa + n_embd_v_gqa)
         * bytes_per_element

bytes_per_element is 2 for the default f16 cache. The “gqa” in those names is the important part: with grouped query attention the number of key/value heads is smaller than the number of query heads, so the cache is narrower than n_embd would suggest. The loader gives you both n_head and n_head_kv, and the ratio between them is the factor by which GQA shrank your cache.

You do not have to trust the arithmetic, because llama.cpp prints the answer once the context is created:

llama_kv_cache: size = 1024.00 MiB (  8192 cells,  32 layers,  1/1 seqs), K (f16):  512.00 MiB, V (f16):  512.00 MiB

“Cells” is the context in tokens, and the K and V halves are itemised separately because they can be quantized independently. If your derived number and this line disagree, believe the line — a model with sliding-window attention or a hybrid recurrent block does not allocate a full cache for every layer, and that is exactly the case the formula above does not cover.

A worked example on an 8B

Take a Llama-3-shaped 8B: 32 layers, n_embd 4096, 32 query heads and 8 KV heads, so head dimension 4096 ÷ 32 = 128 and n_embd_k_gqa = 8 × 128 = 1024, the same for V. Every assumption there is a printed hyperparameter, and the arithmetic is:

per token, per layer : (1024 + 1024) x 2 bytes = 4096 bytes = 4 KiB
per token, 32 layers : 4 KiB x 32              = 128 KiB
at -c 8192           : 128 KiB x 8192          = 1024 MiB
at -c 32768          : 128 KiB x 32768         = 4096 MiB
at -c 131072         : 128 KiB x 131072        = 16384 MiB

So the full trained context of this model costs 16 GiB of cache on top of roughly 4.6 GB of Q4_K_M weights. That is the entire reason a model that “fits in 8 GB” does not fit at long context, and the reason the answer to “why did it load fine yesterday” is almost always that yesterday’s context was shorter. Without grouped query attention — 32 KV heads instead of 8 — every number in that block would be four times larger.

Context costs memory in two other places

The cache is the big one but not the only one.

• The compute buffer. llama.cpp sizes the graph for min(n_ctx, n_ubatch) tokens, so at any context above the micro-batch size this stops growing with context and depends on --ubatch-size instead. Below it, shrinking the context also shrinks the buffer. This is covered in the batch and micro-batch page.
• The logical batch clamp. The context also caps the batch: for a causal model llama.cpp sets n_batch = min(n_ctx, requested batch). Setting a small -c and a large -b does not get you the large batch, and nothing warns you.

Server deployments add a third: -np/--parallel slots divide one cache between concurrent sequences, so the per-request context is the total divided by the slot count rather than the number you typed. A server started with -c 32768 -np 4 gives each request 8192 tokens, and a request that exceeds its slot fails while the load-time log still shows a comfortable-looking 32768 cells. This is the single most common reason a context that worked in llama-cli stops working under llama-server.

What does not grow with context is the weights. That is why the two levers are independent: quantizing harder buys you room for more cache and vice versa, and on a card that is a few hundred megabytes short, dropping the cache from f16 to q8_0 is usually a smaller quality cost than dropping the weights from Q4_K_M to Q3_K_M.

Four ways to make the same context cheaper

• Quantize the cache. -ctk/--cache-type-k and -ctv/--cache-type-v accept f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1, defaulting to f16. Moving both to q8_0 halves the numbers above at a quality cost that is small but is not nothing, and it is heavier on K than on V because keys are what the dot products are computed against.
• Leave flash attention on. -fa/--flash-attn defaults to auto. It does not shrink the cache, but it removes the large intermediate attention matrix from the compute buffer, which is the part that scales with the square of the sequence. See flash attention.
• Ask for the context you need. The cache is allocated up front for the full -c, not grown on demand. A 32k setting for an 8k workload costs four times the memory permanently.
• Do not offload the cache. -nkvo keeps it in system RAM, which trades a large amount of generation speed for device memory. It is a last resort and it is usually worse than reducing context.

None of these changes what the model can attend to. If the context you need exceeds what the model was trained on, that is a different problem with a different flag — see RoPE scaling. And none of them changes the fact that the cache is allocated once, at startup, for the maximum: llama.cpp does not grow it on demand, so there is no configuration in which a long context is free until somebody uses it.