Ollama's num_ctx Parameter and What It Actually Sets

num_ctx is not a limit the model enforces. It is a memory allocation: the number of token slots Ollama reserves in the key-value cache when it starts the runner. Every surprise about it — the model that forgets, the load that succeeds alone and fails under load, the setting that silently does nothing — follows from that one fact.

What the number allocates

When a request arrives, Ollama starts a llama.cpp server process and passes it a context size. That size decides how large a KV cache to allocate up front, before a single token has been processed. The allocation is not lazy and does not grow: a runner started with num_ctx of 32768 has paid for 32,768 token slots whether your prompt is four tokens or thirty thousand.

Two ceilings sit above the number you set. The model’s trained context, recorded in the GGUF metadata, caps it — asking for 256k from a model trained to 32k gets you 32k, because Ollama clamps the request to the value in the file rather than failing. And available memory caps it in a harsher way: if the cache does not fit in VRAM alongside the weights, layers spill to system RAM and the model runs at the speed of the PCIe bus. That spill is silent in the API and visible in ollama ps, whose PROCESSOR column reports a split like 48%/52% CPU/GPU.

The KV-cache arithmetic

The size of the cache is fully determined by numbers a model publishes. For each token, the runtime stores one key vector and one value vector per layer, sized by the number of key-value heads times the head dimension. So:

bytes_per_token = 2 (K and V)
                 x n_layers
                 x n_kv_heads
                 x head_dim
                 x bytes_per_element

total_kv_bytes  = bytes_per_token x num_ctx x num_parallel

Take Qwen3-8B, whose config.json published by Alibaba on Hugging Face gives num_hidden_layers 36, num_key_value_heads 8 and head_dim 128. At f16, two bytes per element:

2 x 36 x 8 x 128 x 2 = 147,456 bytes per token
                     = 144 KiB per token

num_ctx =   4,096  ->    576 MiB
num_ctx =  32,768  ->    4.5 GiB
num_ctx = 262,144  ->     36 GiB

That is arithmetic from a published configuration, not a measurement, and it is why the jump from a 4k default to a 256k default is not a setting you make casually on a laptop. Note what is absent from the formula: the query heads. Grouped-query attention means 32 attention heads share 8 key-value heads on this model, so the cache is a quarter of what a multi-head model of the same width would need. A model withnum_key_value_heads equal to num_attention_heads costs four times as much per token at the same context.

Where the default comes from

Ollama’s Modelfile reference still lists num_ctx with a default of 2048. The server has not used a fixed default for some time. Ollama’s context length documentation states the current behaviour: the default is chosen from detected VRAM, at 4k below 24 GiB, 32k between 24 and 48 GiB, and 256k at 48 GiB and above. The server source uses slightly lower thresholds than the prose — 23 and 47 GiB — with a comment saying the margin exists to absorb small differences in reported totals.

The automatic choice only applies when nobody has expressed an opinion. Setting num_ctx in the request options, or in the model’s own Modelfile, or setting OLLAMA_CONTEXT_LENGTH to anything non-zero, opts that load out of the tier selection entirely. This is the mechanism behind a common report: a user sets 131072 in a Modelfile, the model no longer fits, and the load that used to succeed now spills to CPU — because the automatic sizing that was protecting them is no longer running.

What else moves when you move it

• Concurrency multiplies it. Ollama passes num_ctx x num_parallel as the server’s total context and num_parallel as the slot count, so raising either raises the allocation. Ollama’s FAQ puts it plainly: required memory scales by OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH. See num_parallel and concurrent requests.
• Cache precision halves it. With flash attention active, OLLAMA_KV_CACHE_TYPE=q8_0 stores the cache at roughly half the bytes of the f16 default, and q4_0 at roughly a quarter. Ollama documents this as a global setting affecting every model, and warns that models with high grouped-query ratios lose more precision to it.
• Prefill time grows with what you send, not what you reserve. A large num_ctx costs memory immediately and time only when you actually fill it. Attention work grows faster than linearly in the tokens present, so a genuinely full 128k context is slow for reasons unrelated to the allocation.
• Overflow behaviour is a separate decision. What happens when a conversation exceeds the window — truncate the oldest turns, shift the cache, or refuse — is controlled elsewhere, and is why a context-length error can appear well below the number you set.

Setting it so it sticks

There are four places to set it and they have different lifetimes. Per request, options.num_ctx in the JSON body applies to that load only. In the interactive REPL, /set parameter num_ctx 16384 applies to the session. In a Modelfile, PARAMETER num_ctx 16384 travels with the model to anyone who pulls it. Server-wide, OLLAMA_CONTEXT_LENGTH=64000 ollama serve changes the default for everything and must be set where the server actually reads its environment — a systemd drop-in on Linux, launchctl setenv on macOS, the user environment on Windows. Exporting it in the shell that runs your client changes nothing.

Verify rather than assume. ollama ps prints a CONTEXT column showing the context a loaded model actually got, which is the only number that matters and is frequently not the one you asked for.