Serving Concurrent Requests From One llama.cpp Server

llama-server will serve several requests at once, but not by giving each one its own context. It divides a single KV cache into slots, and the number of slots you ask for divides the context each request gets. Getting this wrong is the most common cause of a local server that quietly answers with less context than you configured.

One context, divided

The --ctx-size (or -c) flag sets a total token budget for the whole server, not a per-request one. The --parallel flag (short form -np) sets the number of slots. The context available to any one request is the first divided by the second.

So -c 32768 --parallel 8 is a server whose requests each get 4,096 tokens. That is not a bug and it is documented behaviour, but it catches people constantly because the flag reads like a per-request limit. llama.cpp reports the result at startup on a line naming n_ctx_per_seq, and it emits a warning when that value is below the model’s trained context. Read that line every time you change either flag.

A slot is more than a share of the cache. Each holds the token sequence it is currently working on, which is what makes prefix reuse possible: if a new request’s prompt shares a prefix with what a slot already holds, the server can skip re-computing that prefix. That is why routing conversations from the same user to the same slot is worth something, and why an eight-slot server under eight unrelated users behaves differently from one under one user.

The arithmetic that decides how many

Slots cost KV cache, and KV cache costs a fixed number of bytes per token set by the model’s architecture. For Llama 3.1 8B — 32 layers, 8 KV heads under grouped query attention, head dimension 128 — one token of fp16 cache is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, or 128 KiB. Work the budget forwards:

Target: 8 slots x 8192 tokens each = 65,536 tokens total

  KV cache  = 65,536 x 131,072 B  = 8.00 GiB
  weights   = Q4_K_M 8B            = 4.58 GiB
  ---------------------------------------------
  subtotal                          12.58 GiB   -> will not fit a 12 GB card

Option A — fewer slots:
  4 slots x 8192 = 32,768 tokens x 131,072 B = 4.00 GiB
  + 4.58 GiB weights = 8.58 GiB               -> fits, with headroom

Option B — quantize the cache (--cache-type-k q8_0 --cache-type-v q8_0):
  8 slots x 8192 = 65,536 tokens x 65,536 B  = 4.00 GiB
  + 4.58 GiB weights = 8.58 GiB               -> same total, twice the slots

The 4.58 GiB is the size the llama.cpp quantize README gives for Q4_K_M on Llama-3.1-8B; substitute the actual size of the file you downloaded, which can differ. The full context-versus-quantization trade works through the choice between the two options. Leave a gigabyte unallocated in either case for the compute buffer, or the server will start and then fail on the first large batch.

What happens to request number nine

It waits. The server keeps a task queue in front of the slots; a request that arrives when every slot is busy is queued and dispatched when one frees, rather than being rejected. From the client’s side that is indistinguishable from a slow response — the connection is open, no error is returned, and time to first token is however long the request ahead of it takes to finish.

That has two consequences that matter for anything with a timeout in front of it. First, latency under load is not degraded smoothly; it is the queueing delay plus the normal generation time, and the queueing delay is dominated by the longest generation currently running, not the average. One request with n_predict set high can hold a slot for minutes. Second, your client timeout is now a concurrency control whether you meant it to be or not — too short and it fails requests that were merely queued.

Continuous batching is what makes the slots worth having. Rather than finishing one request before starting the next, the server steps all active slots forward together, so eight slots generating at once produce far more total tokens per second than eight sequential requests would. What it does not do is make any individual request faster; per-request rate under eight-way load is lower than under one-way load, because they are sharing the same memory bandwidth. Total throughput up, single-request latency down — the standard throughput-versus-latency trade, here with a knob on it.

Building it

1. Compute the budget before you start anything. Take the file size of your GGUF, add slots x per-slot-context x KV bytes per token, add a gigabyte, and check it against the free VRAM in nvidia-smi. The KV figure comes from the model’s config: 2 x layers x kv_heads x head_dim x bytes.
2. Start the server with both flags set explicitly. Never rely on the defaults for these two, because they have moved:

   llama-server \
     -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
     -c 32768 \
     --parallel 4 \
     -ngl all \
     --host 127.0.0.1 --port 8080

3. Read the startup log for n_ctx_per_seq and confirm it is the 8,192 you intended and not something smaller. If it is smaller, one of your two flags is not what you think it is.
4. Confirm the same figure over the API so that your client and your server agree:

   curl -s localhost:8080/props | python3 -m json.tool | grep n_ctx

5. Fire more requests than you have slots and confirm the extras complete rather than erroring:

   for i in $(seq 1 9); do
     curl -s -o /dev/null -w "req $i: %{time_total}s\n" \
       localhost:8080/completion \
       -d '{"prompt":"Count to twenty.","n_predict":120}' &
   done
   wait

   With four slots, the last five completion times should cluster noticeably above the first four. That step function is the queue.

Watching the slots while it runs

The /slots endpoint is enabled by default and shows each slot’s state and the prompt it is holding. That is genuinely useful for debugging and it is also a reason to think about who can reach the port — anyone who can query it can read the prompts of everyone else using the server. --no-slots turns it off, and binding the server to a network interface is a decision to make with that in mind.

watch -n 1 'curl -s localhost:8080/slots | python3 -c "
import json,sys
for s in json.load(sys.stdin):
    print(s[\"id\"], s.get(\"state\"), s.get(\"n_ctx\"))
"'

One last thing that surprises people: slots do not resize. If you sized for eight and are usually serving one, that one request is still limited to an eighth of the context, with the other seven eighths allocated and idle. There is no dynamic reallocation, so the slot count is a decision about your worst case that you pay for in your common case. Pick it from measured concurrency, and if you do not have measured concurrency, pick two.