Sliding Window Attention and Streaming Long Documents

Full attention makes every token look at every earlier token, which is quadratic and eventually unaffordable. A sliding window makes every token look at only the last W, which is linear and eventually forgetful. The interesting engineering is entirely in what you add back.

The arithmetic that motivates it

Attention cost is the number of query–key pairs. For a sequence of length n with full causal attention that is about n²/2; with a window of W it is about n×W, once n exceeds W.

n = 128,000 tokens

full window:     128,000 x 128,000 / 2  = 8.19e9 pairs
W = 4,096:       128,000 x 4,096        = 5.24e8 pairs

                 ~15x less attention work, per head, per layer

KV cache, which is the memory side of the same story:
full:            grows with n, without bound
windowed:        capped at W entries, forever

The second half of that is often the more important one. The KV cache is what limits how many concurrent sequences a server can batch, and an unbounded cache is what makes very long context expensive to serve even when the arithmetic is affordable. A bounded window bounds the memory, which is why streaming architectures use it.

Three published designs

• Longformer (Beltagy, Peters and Cohan, 2020) combined a sliding window with a small set of global tokens that attend to everything and are attended by everything — typically the task’s question or a classification token. Local cost plus a thin global channel, which is a pattern that keeps reappearing.
• Mistral 7B (Jiang et al., 2023) applied a 4,096-token sliding window at every layer and made the stacking argument explicitly: because layer k’s window sees representations that already summarise layer k−1’s window, information propagates about W tokens per layer, so a 32-layer model has a theoretical attention span in the region of 131,000 tokens despite never computing an attention score across that distance.
• Hybrid layer stacks. Several recent families interleave a majority of local-attention layers with occasional full ones. The full layers restore genuine long-range lookup; the local ones keep the average cost near-linear. It is a better trade than either extreme and it is why “does this model use sliding window attention” is usually a per-layer question.

Why the first few tokens must stay

The naive rolling window fails in a way that is genuinely surprising. Xiao, Tian, Chen, Han and Lewis, Efficient Streaming Language Models with Attention Sinks (2023), observed that if you evict the very first tokens of a sequence as the window slides, perplexity explodes — not degrades, explodes.

Their explanation is that models learn to dump surplus attention mass on the first few positions. Softmax weights must sum to one even when no earlier token is relevant, and the initial tokens become the sink for that excess. Remove them and the distribution is forced onto tokens that were never meant to carry it. Their fix is almost embarrassingly small: keep the first handful of tokens permanently, slide the window over everything else, and a model can then stream millions of tokens with stable perplexity.

It is worth knowing this even if you never implement it, because it is the mechanism behind an implementation-level bug you may meet: a serving stack that trims the beginning of a KV cache to make room will produce output that degrades sharply and inexplicably.

A window is not a memory

The distinction the StreamingLLM authors were careful about is the one to carry away. Stable perplexity over four million tokens means the model keeps producing fluent, coherent text. It does not mean the model can answer a question about token 1,000 when it is at token 3,000,000. That information left the window and is gone.

So sliding window attention buys unbounded fluency at bounded cost, not unbounded recall. If your task needs recall, you need either genuine long context — with the caveats on effective context length — or a retrieval layer that puts the old material back into the window when it becomes relevant.

When a window beats a bigger context

The general rule: if relevance decays with distance, a window is not a compromise but the right model of the problem. If it does not, no amount of window engineering will substitute for having the tokens present.

Two things to check if you are running the weights yourself. First, whether the model uses a window at all: open-weights configs expose it as a sliding_window field, sometimes alongside a layer pattern saying which layers are local and which are full. Second, whether your serving stack implements it. A window is only a saving if the runtime actually stops keeping the evicted KV entries; several inference servers historically accepted a windowed model and kept the full cache, which is correct output at none of the promised memory benefit.

The other interaction worth anticipating is with prompt caching. A cached prefix is a stored KV block, and a rolling window is a policy for discarding KV blocks. They can be made to coexist, but the naive combination — cache a long prefix, then slide past it — throws away exactly what you paid a write premium to keep. Where both are available, the prefix wants to be inside the retained region, which in the attention-sink design means at the very start of the sequence.