Serving Parallel Requests With llama.cpp's Server Slots

A llama.cpp server slot is a seat at the model: one conversation’s KV cache, one position counter, one set of sampler state. The number of seats and the length of each are the same budget, and --parallel is how you split it.

What a slot is

The server holds one model and serves many requests against it by giving each in-flight request a slot. A slot owns a region of the KV cache and the bookkeeping that goes with it; continuous batching, on by default, then interleaves tokens from every busy slot into shared forward passes, which is what makes concurrency worth having at all — generation at batch size one leaves the arithmetic units idle, and filling them with a second and third sequence is nearly free.

-np, spelled --parallel, sets the number of slots. Its default is -1, meaning auto. A request arriving when every slot is busy is not rejected; it is deferred until one frees, which is visible as latency rather than as an error, and is counted separately in the Prometheus output as llamacpp:requests_deferred.

How the context is divided

-c is not the context each conversation gets. It is the total, and llama.cpp derives the per-sequence figure from it in src/llama-context.cpp:

if (cparams.kv_unified) {
    cparams.n_ctx_seq = cparams.n_ctx;
} else {
    cparams.n_ctx_seq = cparams.n_ctx / cparams.n_seq_max;
    cparams.n_ctx_seq = GGML_PAD(cparams.n_ctx_seq, 256);
}

In the non-unified case that is a plain division by the number of sequences, padded up to a multiple of 256. So -c 32768 -np 4 gives four slots of 8192 tokens each, and a request that needs a 12k prompt fails on a server whose advertised context is 32k. That surprise is the single most common misreading of these two flags.

The padding has a consequence too. Because n_ctx is recomputed as n_ctx_seq × n_seq_max afterwards, an indivisible pair gets adjusted and llama.cpp says so: n_ctx is not divisible by n_seq_max - rounding down to N. You will also see either n_ctx_seq (X) < n_ctx_train (Y) -- the full capacity of the model will not be utilized or the corresponding overflow warning, and those two lines are the fastest confirmation that the arithmetic landed where you intended.

The unified buffer changes the sum

The branch above is the part to read twice. With --kv-unified — one KV buffer shared across all sequences — n_ctx_seq is set to the whole of n_ctx, and slots draw from a common pool rather than each owning a fixed slice. That is better when your traffic is uneven, because one long conversation can use most of the cache while three short ones use very little, instead of every slot reserving a quarter it may never touch.

The flag pair is -kvu, --kv-unified and -no-kvu, --no-kv-unified, and the documented default is enabled when the number of slots is auto. So the answer to “does -c divide” is: it depends on which mode you are in, and the log line naming n_ctx_seq is the authority for your run. Shared pooling is not free — sequences now contend for capacity, so a long one can crowd out a short one, and the failure mode moves from “deterministic per-slot limit” to “depends what else is running”.

Watching it at runtime

GET /slots, enabled by default, returns one object per slot with its id, its n_ctx, whether it is processing, the sampling parameters actually in force and a next_token block with n_decoded and n_remain. Reading n_ctx off that response is the direct way to check the division above rather than reasoning about it.

Two further fields on that response matter for capacity work. A request may name id_slot to pin itself to a particular slot, defaulting to -1 for “any idle one”, which is occasionally useful and usually a mistake: pinning defeats the scheduler and leaves the other slots idle. And because prompt caching is per slot, which slot a conversation lands on decides whether its prefix is still cached — the reason a second turn is sometimes fast and sometimes not under load. That interaction is the subject of the prompt caching page.

curl -s http://127.0.0.1:8080/slots | jq '.[] | {id, n_ctx, is_processing}'

# and the aggregate view
curl -s http://127.0.0.1:8080/metrics | grep -E 'busy_slots|requests_deferred'

llamacpp:n_busy_slots_per_decode is the number that tells you whether your slot count is doing anything: if it sits near 1 while requests are being deferred, the bottleneck is not concurrency, and if it sits near your slot count then adding slots will help until memory stops you.

Sizing the two numbers together

Start from the memory, not from the concurrency you want. The KV cache is the part that scales with both numbers, and in the non-unified case the total is fixed by -c alone — the split changes who may use it, not how much there is. What changes with slot count is the compute buffer and the scheduling, not the cache size, so the honest question is which shape of failure you prefer: a per-slot limit that rejects long prompts predictably, or a shared pool where a long conversation degrades everyone.

Then set the slot count from the longest prompt you must accept. If that prompt is 16k and you are not running unified, -c must be at least 16k times the slot count. If that product does not fit, --parallel is where to give ground, because a deferred request waits and a truncated one is wrong. The per-token memory arithmetic behind “does this fit” is worked through in the context-size page, and the batch flags that decide how tokens from several slots are packed into a pass are in the batch and ubatch page.