What Happens When You Exceed a Self-Hosted Llama's Context Length

Three widely used local servers do three different things with a prompt that does not fit, and only one of them is loud about it. Working out which you are looking at is most of the fix.

The messages you are looking at

vLLM, at request time. An OpenAI-compatible 400 whose body says roughly:

This model's maximum context length is 8192 tokens. However, you
requested 8500 tokens (8000 in the messages, 500 in the completion).
Please reduce the length of the messages or completion.

This is the most useful error of the three, because it decomposes the number: prompt tokens, requested completion tokens, and the limit. Note that the limit quoted is the served limit, which is whatever --max-model-len was set to and may be far below the checkpoint’s documented window.

vLLM, at startup. A different failure with the same root cause — the engine refusing to start because it cannot fit the model’s full window in the memory left over after the weights:

ValueError: The model's max seq len (131072) is larger than the maximum
number of tokens that can be stored in KV cache (32768). Try increasing
gpu_memory_utilization or decreasing max_model_len when initializing
the engine.

That message is telling you the memory arithmetic did not work out. The KV cache figure is derived from free VRAM, and the resolution is almost always to set --max-model-len to something you actually need rather than to chase the full 128K — see the KV cache arithmetic for why the full window is so expensive.

llama.cpp. The server rejects the request with a message about the context size, in the shape of the request exceeds the available context size, try increasing the context size. Separately, at load time, it prints a warning if the context you asked for exceeds what the model was trained on. The controlling flags are -c / --ctx-size for the total and -n for prediction length.

Ollama. Frequently, nothing. The server truncates the prompt to fit and answers anyway; the evidence is a line in the server log about truncating the input, and an answer that has quietly forgotten the beginning of your conversation.

Input and output share one budget

The single most common misreading of these errors is treating the context length as a limit on the prompt. It is a limit on the sequence, and the sequence includes everything the model generates. If the served window is 8192 and you send 8000 tokens, you have 192 left to answer in, and requesting 500 is an error before a token is produced.

This is why vLLM reports both numbers and adds them. The check is:

prompt_tokens + max_tokens <= max_model_len

And the prompt tokens are not just your text. The chat template adds header and separator tokens for every message, a system message you did not write may have been injected, and each turn of history is counted again on every request because there is no server-side state. A conversation that worked for twenty turns and fails on the twenty-first has not hit a bug; it has been growing the whole time. The template overhead is itemised in the chat template tokens page.

The dangerous case: no error at all

Two stacks will take an over-long input and give you an answer.

Ollama sets a context size per model and truncates to fit. The number comes from the num_ctx parameter, which historically defaulted far below what modern Llama checkpoints support — a model documented at 128K served with a default context in the low thousands is the usual shape of the surprise. Override it in the request options or bake it into a Modelfile:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [{"role": "user", "content": "..."}],
  "options": { "num_ctx": 32768 }
}'

Raising num_ctx allocates more KV cache, so it costs memory and can push the model off the GPU into system RAM, which is slow rather than broken. That trade is the reason for the conservative default.

Raw transformers is the other silent case, and the worst, because it has no serving layer to check anything. Passing a sequence longer than the position embeddings support gets you either an index error deep in the attention implementation or — with a model using scaled rotary embeddings, which every Llama 3.1 and later checkpoint does — no error and steadily degrading output. Nothing tells you the model has stopped attending usefully to the far end of the prompt.

The signature of silent truncation is worth memorising: the model answers coherently but ignores the earliest part of the input. If a document summary is systematically missing the opening, or an agent has forgotten its system prompt, suspect truncation before you suspect the model.

Fixing it

1. Find the served limit, not the documented one. vLLM prints max_model_len at startup and reports it in the error; llama.cpp logs n_ctx when it loads; Ollama shows the value in the model’s parameters. The model card is not the authority here — the process is.
2. Count your prompt with the model’s own tokenizer. Use tokenizer.apply_chat_template(messages, tokenize=True) and take the length, so that template tokens are included. A word-count estimate is not close enough at the boundary.
3. Raise the limit if memory allows. --max-model-len for vLLM, --ctx-size for llama.cpp, num_ctx for Ollama. Expect to trade concurrency for it: the same VRAM serves fewer simultaneous requests at a longer window.
4. Or reduce what you send. Truncate history from the oldest turn, summarise older turns into one message, or retrieve the relevant part of a document rather than pasting all of it. Dropping the middle and keeping the ends is usually better than dropping the end, because the system prompt and the current question both matter more than turn nine.
5. Reserve room for the answer explicitly. Compute max_tokens = max_model_len - prompt_tokens - margin instead of hardcoding it, and fail your own way if that number is too small to be useful.

The fix that is not a fix

Searching this error leads quickly to advice about overriding the position-encoding scale — vLLM’s --rope-scaling override, llama.cpp’s --rope-freq-scale, or editing max_position_embeddings in a local copy of config.json. It does stop the error. That is the only thing it reliably does.

The mechanism is worth being clear about, because the advice is not always wrong. A Llama 3.1 or later checkpoint already ships a scaling configuration that Meta trained for; overriding it to reach beyond the documented window asks the model to attend at positions it never saw, and quality falls off in the way that positions being out of distribution predicts. Nothing errors. The model produces text, and the text gets worse in a way that is hard to attribute later.

The two cases to separate:

• Restoring a window the checkpoint documents — for instance a runtime that failed to read rope_scaling and defaulted to 8,192 on a 128K model. Here the override is a genuine fix, and it is worth checking your runtime’s version support before assuming anything else.
• Extending past what the checkpoint documents — an experiment, not a deployment. If you do it, evaluate at the extended length rather than assuming, and expect the degradation to be gradual rather than obvious.

Editing config.json in a local copy has a second cost worth noting: it silently un-pins you from the published checkpoint, so nobody reproducing your setup from the repository name gets the same model. If you must do it, do it in code with an explicit override rather than by editing a downloaded file.

Preventing it

The durable fix is a check in your code before the request leaves, because the error is cheap to produce locally and expensive to discover in production. Two habits are worth more than the rest:

• Log prompt token counts on every request. A slow climb over days is visible in a graph and invisible in an incident.
• Set the served window explicitly at startup. Never rely on a default, in any of the three servers. An explicit --max-model-len 32768 is a number you chose and can reason about; a default is a number that changes when you upgrade.