KoboldCpp's Context Shifting Explained

In a long conversation at a fixed context size, something has to give when the window fills. The naive answer is to drop the oldest messages and reprocess what remains, which costs a full prefill on every turn. Context Shifting is the alternative: move the cache rather than rebuild it.

What a full context costs without it

Generation and prompt processing are different operations with different costs. Prefill reads every token of the prompt in one pass and is bounded by arithmetic; generation produces one token per pass and is bounded by memory bandwidth. On local hardware the prefill rate is typically many times the generation rate, which is exactly why reprocessing feels like a stall rather than being invisible: you are paying for thousands of tokens of prefill before the first token of a reply appears.

The shape of the cost is the part to reason about. If your window is C tokens and each turn adds n new ones, a naive drop-and-reprocess pays prefill on roughly C tokens per turn while the useful work is n. The waste ratio is C / n, so it gets worse as you raise the context size and worse as the messages get shorter — the two things people do most. Attention is quadratic in sequence length on top of that, so doubling C more than doubles the reprocess.

Do not take a number for this from anywhere, including here. Your own runtime prints its prompt evaluation rate and its generation rate on every request; those two numbers, from your machine and your model, are the only ones that mean anything. Divide the context size by the prompt rate and you have the stall you are avoiding.

What shifting actually moves

The KV cache stores a key and a value vector for every token, in every layer. It is the reason generation does not re-read the whole conversation for each new token, and it is a plain array indexed by position.

When the window fills, KoboldCpp does not throw that array away. It removes the entries for the oldest tokens and shifts the remaining entries down to close the gap, leaving room at the end for new ones. The project’s wiki describes it as using KV cache shifting to remove old tokens and add new ones without requiring any reprocessing. Everything still in the cache was already computed; nothing about those tokens has changed except where they sit.

A complementary feature does the other half. Fast Forwarding, on by default and disabled with --nofastforward, detects that the start of this turn’s prompt matches what was processed last turn and skips it, processing only the genuinely new tokens. Shifting handles the window overflowing; fast forwarding handles the ordinary case where it has not.

Why the positions have to be rewritten

Moving cached entries is not a memory copy alone, and this is the part that explains the feature’s limits. Modern models encode position rotationally: the key vectors carry a rotation applied according to where the token sat in the sequence. A key computed at position 4,000 is not the same vector as the same token at position 3,000.

So when tokens shift down by some amount, their cached keys must have that rotation adjusted to match their new positions — a shift operation applied across the cache. That is cheap: it touches the cache once rather than running the model. But it is only defined for models whose position encoding can be adjusted this way. Architectures that encode position differently, or that keep state rather than a per-token cache, cannot be shifted at all, which is why this is a GGUF feature tied to what the underlying runtime supports rather than a universal one.

What turns it off without telling you

Shifting only works while the surviving cache is still valid for the prompt you are about to send. Anything that changes a token in the middle of the context invalidates everything after it, because those later entries were computed with the old token in view. The wiki is specific about the cases:

• Editing memory or earlier story text. A change near the start of the context invalidates the whole cache behind it and you pay a full reprocess. This is why editing an old message feels expensive and appending a new one does not.
• World Info. Entries are injected into the context based on what has been said, so the injected block changes between turns. A block that changes in the middle of the prompt breaks the match that shifting depends on.
• --noshift — the explicit off switch.
• Non-GGUF models. The feature is GGUF-only.

There is a fallback: SmartContext, the older approach, which reserves roughly half the context as a spare buffer so it only reprocesses on every second overflow rather than every one. Context Shifting overrides it where both are enabled, and is better precisely because it costs no context space at all. If you are running a model where shifting is unavailable, SmartContext is what you have left.

One honesty note about what shifting is doing to your conversation: the oldest tokens are gone. The model is not summarising them or compressing them, it is dropping them, so a long chat is a rolling window over its own history. That is the same trade discussed on LM Studio’s overflow policies, reached by a different route.

llama.cpp made the opposite default

KoboldCpp enables Context Shifting by default and gives you --noshift to turn it off. Upstream llama.cpp’s server made the opposite choice: its README documents --context-shift, --no-context-shift with the default listed as disabled, so a plain llama-server does not shift unless you ask.

That divergence is worth knowing if you are comparing the two, because it produces exactly the symptom people report as “llama.cpp is slower in long chats”. It is not slower at generating; it is doing a different thing at the window boundary. The reason upstream is conservative is the honesty note above — silently discarding the start of a context changes results in ways a server operator may not want by default, and preferring an explicit failure to a quiet truncation is a defensible position for a general-purpose server.