Prompt Caching in llama.cpp's Server

A long system prompt is re-read from scratch on every request unless something keeps its keys and values around. llama.cpp has two separate mechanisms for that, with confusingly similar names, and only one of them is on the server.

Two caches with similar names

--prompt-cache FNAME is a file. It belongs to llama-completion, where llama.cpp registers it, and it saves model state to disk so a later invocation of the same one-shot binary starts faster. It is not a llama-server flag, and looking for it there is a common way to lose twenty minutes.

The server’s mechanism is --cache-prompt, and it is enabled by default. It keeps the KV cache of the previous request in a slot and reuses whatever prefix the next request shares with it. Around it sit --cache-reuse N, -cram / --cache-ram N for how much memory the cache may occupy, and the slot save and restore endpoints. The rest of this page is about those, because that is what is running when a second turn of a conversation comes back faster than the first.

What the server reuses, and when

The unit of reuse is a prefix, and the reason is positional. Each entry in the KV cache is the key and value computed for a token at a position, conditioned on everything before it. Tokens 0 to k of the new request can reuse the cached entries only if they are the same tokens in the same order as last time; from the first difference onwards, every subsequent entry was computed against a context that no longer applies.

The server documents the effect of cache_prompt exactly this way: the common prefix does not have to be reprocessed, only the suffix that differs. It also documents the cost, which is worth quoting to anyone chasing a reproducibility bug — because logits are not guaranteed bit-for-bit identical across different batch sizes, and a cache hit changes how many tokens are processed in the prefill batch, enabling the option can make results nondeterministic.

You do not have to infer whether it worked. The timings object on the response reports cache_n, the number of prompt tokens reused from cache, next to prompt_n, the number actually processed, and the documentation states that the total context is prompt_n + cache_n + predicted_n.

curl -s http://127.0.0.1:8080/completion \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"<your long system prompt>\n\nQuestion: ...","n_predict":64}' \
  | jq '.timings | {cache_n, prompt_n, prompt_ms, prompt_per_second}'

What throws the cache away

Everything that changes an early token, and that is a longer list than people expect.

• A timestamp or a session id at the top of the system prompt. One differing token at position 12 invalidates positions 12 onwards, which is the entire prompt. Move volatile material to the end and the prefix survives.
• Reordered blocks. Retrieved chunks concatenated in whatever order the vector store returned them will differ between requests that fetched the same documents. Sort them.
• A different slot. The cache lives in the slot, so a request routed to another slot starts cold. Under concurrency this is why the same request is sometimes fast and sometimes not — see how slots divide the server.
• A chat template change. The template is what turns messages into tokens on the chat completions endpoint, so a build that renders the role header differently invalidates every cached prefix you had.

--cache-reuse N softens the first rule. Instead of giving up at the first divergence, the server will attempt to reuse chunks of at least N tokens that appear after it by shifting their positions in the KV cache; it defaults to 0, which is off, and it requires prompt caching to be enabled. It is the flag to reach for when your prompts differ in the middle rather than at the end.

How much prefill it saves

The saving is prefill time, and prefill time is tokens divided by a rate you measure rather than one anybody can quote for you. Get the rate from prompt_per_second in the response you just made, or from a pp row of llama-bench. Call it R tokens per second. Then for a system prompt of S tokens followed by a question of Q tokens:

cold   prefill = (S + Q) / R
warm   prefill = Q / R          (cache_n = S, prompt_n = Q)
saved          = S / R

# worked example, R taken from your own log:
# S = 2000, Q = 40, R = 900 tok/s
#   cold = 2040/900 = 2.27 s
#   warm =   40/900 = 0.04 s
#   saved                = 2.22 s per request

The shape of that result is the point: the saving is proportional to the shared prefix and independent of the answer length, so caching is worth most exactly where time to first token dominates — a long instruction block with short questions against it. It does nothing for generation speed, because generation was never the part being repeated.

Substitute your own R. Prompt-processing rates differ by more than an order of magnitude across backends, quantisations and context depths, and any number quoted without the hardware attached is decoration.

The --prompt-cache file, and the slot endpoints

For the one-shot binary, --prompt-cache FNAME writes the state after the prompt to a file and reuses it next time; --prompt-cache-all extends that to inputs and generations, and --prompt-cache-ro reads without updating. llama.cpp attaches a caveat worth repeating: restoring a cached prompt does not restore the exact session state, so even with a fixed seed you are not guaranteed the same token sequence as the original run.

The server’s equivalent of persistence is per slot. With --slot-save-path PATH set, POST /slots/{id}?action=save writes that slot’s prompt cache to a named file under that directory and returns n_saved and a save_ms timing; action=restore loads it back, reporting n_restored; and action=erase clears the slot and reports n_erased. That is the mechanism for warming a fixed system prompt into a slot at start-up rather than paying for it on the first user request.