When a Local Model Silently Truncates Context You Thought It Kept

The model answers as if the top of your prompt was never there. No error came back, the request returned 200, and the system prompt you carefully wrote is being ignored. If you look in the server’s own log rather than at the response, you will usually find one of these two lines.

# Ollama, in the server log — a WARN, not an error, and never in the response:
level=WARN source=runner.go:129 msg="truncating input prompt" \
  limit=2048 prompt=3858 keep=5 new=2048

# llama.cpp, returned as HTTP 400 — this one you do see:
"the request exceeds the available context size.
 try increasing the context size or enable context shift"

The lines that tell you it happened

Those two are the whole difference between a runtime that fails loudly and one that fails quietly, and it is worth being clear about which you are running.

llama.cpp’s server rejects the request. Context shift — the behaviour where the oldest tokens are discarded to make room — is disabled by default, and with it disabled an over-long prompt is a 400 with the string above. That is the correct design: an over-long prompt is a bug in the caller and silently answering a different question is worse than failing.

Ollama truncates. The prompt is trimmed from the start so it fits num_ctx, a warning goes to the server log, and the API returns a perfectly normal response. A caller who never reads the server log has no way to know. Because the trim is from the start, the first thing discarded is the system prompt, which is why the failure so often presents as “the model stopped following its instructions” rather than as anything about length.

Why some of it is silent

Truncating rather than erroring is a defensible choice for an interactive chat tool: a long conversation should keep working rather than hitting a wall, and dropping the oldest turns is the obvious way to do that. The problem is that the same runtime is also an API server, and a programmatic caller has no equivalent of the user noticing that the conversation lost its memory.

The same argument applies to llama.cpp with --context-shift explicitly enabled. Turning it on converts a loud failure into a silent one by design, so enable it for an interactive session and leave it off for anything a program calls.

Four places the low ceiling comes from

The truncation is a symptom. The ceiling is set in one of four places, and they need different fixes.

• The runtime’s default is lower than the model’s. Ollama’s default context length has historically been small relative to what modern models are trained for, and recent versions choose it based on available VRAM rather than from the model. A model advertised at 128k served at 4k is not a contradiction; it is a default. Set OLLAMA_CONTEXT_LENGTH on the server, or num_ctx in the request options, explicitly.
• Your context is divided by your slot count. llama-server’s --ctx-size is a total across all slots, so -c 32768 --parallel 8 gives each request 4,096 tokens. The startup log names the result as n_ctx_per_seq and warns when it is below the model’s trained length. The slot arithmetic is a page of its own.
• Your client is not forwarding the setting. This is the nastiest one. An OpenAI-compatible endpoint has no field for num_ctx, so a client written against the OpenAI schema and pointed at a local server cannot set it, and the server falls back to its own default no matter what you configured in the client. Set it server-side and stop trying to send it per-request.
• The GGUF metadata says something you did not expect. The trained context length is stored in the file, and a quantizer can override metadata keys. Check what your file actually claims with gguf-dump before assuming the runtime is at fault.

Detecting it without reading logs

Two checks, neither of which requires log access. The first is to ask the server what it thinks its context is:

# llama.cpp
curl -s localhost:8080/props | python3 -m json.tool | grep -i n_ctx

# Ollama — the running model's actual loaded context
ollama ps

The second is to compare what you sent against what the server says it evaluated. llama.cpp returns tokens_evaluated in its completion response and Ollama returns prompt_eval_count; either one, when it is smaller than the token count of the prompt you sent, is truncation caught in the act.

curl -s localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "<a prompt you know is 6000 tokens>",
  "stream": false
}' | python3 -c 'import json,sys; d=json.load(sys.stdin); \
print("evaluated:", d["prompt_eval_count"])'
# If this prints 4096 for a 6000-token prompt, you have your answer.

A third check works when you cannot instrument anything: put a canary at the very top of the prompt — a nonsense token, or an instruction to begin every reply with a specific word — and ask the model to repeat it. Because truncation is from the start, a canary at position zero is the first casualty. It is crude and it is conclusive.

Fixing it for good

1. Set the context explicitly on the server, always. OLLAMA_CONTEXT_LENGTH=16384 ollama serve, or llama-server -c 16384 --parallel 1. Never rely on a default for this value; the defaults have changed more than once and will again.
2. Check the arithmetic will fit before you raise it. Context costs VRAM at a fixed rate per token — 128 KiB per token at fp16 for Llama 3.1 8B, derived from its layer and head counts. 16k tokens is 2 GiB. Ask for more than the card has and the server fails at warmup, which is at least a loud failure.
3. Leave context shift disabled for programmatic callers. You want the 400. Handle it in the caller by summarising or truncating deliberately, where you control what gets dropped.
4. Count tokens on the client before sending. The only robust fix is to know the size of your prompt with the model’s own tokenizer and to make the trimming decision yourself. Context window and max tokens are different budgets and both have to fit: the prompt plus the reply you asked for cannot exceed the window.
5. Assert on the returned count. Compare prompt_eval_count against your own token count on every request in development, and log a warning when they differ. This is three lines of code and it converts a class of silent failure into a visible one permanently.