Ollama's num_parallel Setting for Concurrent Requests

Raising num_parallel does not load the model twice. It divides one loaded model into several request slots that share the weights and do not share the cache — which is why the memory cost is real but not proportional to the model size, and why the arithmetic surprises people in both directions.

Slots, not processes

When Ollama starts a runner it passes two arguments that matter here: a total context size, and a number of parallel slots. The llama.cpp server divides its context allocation evenly among the slots, and each in-flight request occupies one. The weights are loaded once and read by every slot, so the second concurrent request costs no additional copy of the model.

This is also a throughput mechanism rather than only a capacity one. Generating one token for one request reads the whole active weight set from memory to do a very small amount of arithmetic, which leaves a GPU almost entirely idle in compute terms. Serving several sequences from one weight read is what batching buys: total tokens per second across all requests rises well beyond what a single stream achieves, even though each individual stream may get slightly slower.

What it multiplies

The exact relationship is visible in what Ollama hands the runner: the context flag is set to num_ctx x num_parallel and the slot count to num_parallel. Ollama’s FAQ states the same thing from the other direction — required memory scales by OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH — and gives the worked case: a 2K context with 4 parallel requests results in an 8K context and additional memory allocation.

So each slot gets a private KV cache of num_ctx tokens, and the total cache is the product. Using the per-token figure derived on the num_ctx page for Qwen3-8B — 144 KiB per token at f16, from 36 layers, 8 key-value heads and a head dimension of 128:

num_ctx 8,192, num_parallel 1  ->  8,192 x 144 KiB   = 1.13 GiB
num_ctx 8,192, num_parallel 4  -> 32,768 x 144 KiB   = 4.50 GiB
num_ctx 8,192, num_parallel 8  -> 65,536 x 144 KiB   = 9.00 GiB

weights (q4_K_M)                                     ~ 4.87 GiB

On an 8 GB card, that model at four slots is already asking for more than the card holds once weights are counted, and the load will spill to system RAM rather than refuse. The failure is a performance collapse, not an error, which is what makes it hard to attribute.

One consequence catches people who never intended to serve concurrency at all: the multiplication happens at load time, not when a second request arrives. A runner started with four slots has allocated four caches before the first request finishes, so raising the setting on a machine that only ever sees one user at a time costs three quarters of the cache budget for nothing. This is the reverse of the intuition that concurrency settings are free until used.

The other half of the interaction is the automatic context sizing. If you have not set a context anywhere, Ollama chooses one from detected VRAM, and that choice is made without reference to num_parallel — the multiplication is applied afterwards. On a card in the 32k tier, four slots therefore ask for 128k of cache. That is exactly the configuration that loads fine on a quiet machine and spills the moment it is used properly, and it is the reason to set num_ctx explicitly whenever you set slots.

Budgeting the split

Rearranged, the setting answers a budgeting question:

kv_budget    = free_vram - weights - overhead
tokens_total = kv_budget / bytes_per_token
num_ctx      = tokens_total / num_parallel

Which makes the trade explicit. A fixed cache budget buys you either one long conversation or several short ones, and the product is what is constant. Four slots of 8k and one slot of 32k cost the same memory. That is the decision to make deliberately: a coding assistant wants the long single context, an API serving many short classification calls wants the slots.

The current default is 1 in Ollama’s configuration source, and the FAQ documents the same value. This has not always been true — earlier releases picked 4 or 1 automatically from available memory — so a guide that describes automatic selection is describing a different version than the one you are running. Check with ollama serve --help, which prints the environment variables and their effective values.

What happens to the requests that do not fit

Requests beyond the slot count are queued rather than rejected, up to OLLAMA_MAX_QUEUE, which Ollama’s FAQ documents as defaulting to 512. Past that the server returns HTTP 503 to say it is overloaded. A queued request is not slow to generate; it is slow to start, which shows up as time-to-first-token climbing while the per-token rate stays flat. Those two numbers coming apart is the signature of queueing rather than of a slow model, and averaging them into one latency figure hides it completely.

Loading several different models at once is a separate axis, governed by OLLAMA_MAX_LOADED_MODELS — documented as defaulting to three per GPU — and it costs a full copy of each model’s weights rather than sharing them. That interaction is covered in running multiple models simultaneously.

Choosing a number

• One user, one machine. Leave it at 1. Slots you do not use still divide the context allocation.
• A small team or an internal service. Two to four slots, with num_ctx reduced to keep the product inside VRAM, usually beats one slot and a queue for perceived responsiveness.
• Batch work with no human waiting. Slots buy real throughput here, because nobody minds each item being slightly slower if the whole set finishes sooner.
• Any change at all. Confirm with ollama ps afterwards that the model still reports 100% GPU. A setting that pushed the model into system RAM will look like a regression everywhere and be attributed to anything but the slot count.