Gemma 2's Sliding Window Attention and Its Effect on Long Context

Half of Gemma 2’s attention layers cannot see more than 4,096 tokens back. That is not a bug or a degradation mode; it is the architecture, and it is the reason a 27B model fits where it does.

The alternating pattern

Gemma 2 interleaves two kinds of attention layer. Local layers use a sliding window of 4,096 tokens: a query at position i attends only to keys in roughly [i - 4096, i]. Global layers attend over the entire sequence up to the window limit of 8,192. The layers alternate, one local then one global, through the depth of the model.

Both numbers are in the checkpoint config, and the pattern is expressed there too:

from transformers import AutoConfig

cfg = AutoConfig.from_pretrained("google/gemma-2-9b-it")
print(cfg.sliding_window)             # 4096
print(cfg.max_position_embeddings)    # 8192
print(cfg.num_hidden_layers)

In the transformers implementation the alternation is decided per layer index, so an even layer and an odd layer genuinely run different masks. If you are reading the model code and expecting one uniform attention implementation, that is the thing to look for.

The idea is not new — sliding-window attention predates Gemma by years, and Mistral shipped a version of it — but the interleaving is the part that matters. A model with sliding-window attention in every layer has a hard information horizon: nothing can reach further than the window times the depth, and in practice much less. Keeping every other layer global means the horizon is the full context, and only the bandwidth across it is reduced. That is a much gentler degradation than a uniform local model, and it is why the alternating design has been copied rather than the pure one.

What it saves in the KV cache

During generation, every attended-to token has to have its keys and values kept in memory, per layer, per attention head. That cache is usually the thing that decides how many concurrent conversations a GPU can hold, and it grows linearly with sequence length.

With alternating attention, only the global layers need a cache that grows to the full 8,192. The local layers can evict anything older than their 4,096-token window, because nothing in the layer will ever look at it again. At full context that is half the layers holding half as much, so the cache is about three quarters of what a fully global model of the same shape would need. Work the arithmetic on a fully extended conversation:

# Relative KV cache at full 8192-token context.
# Units are arbitrary; only the ratio matters.
global_layers = 0.5 * 8192   # full-length cache
local_layers  = 0.5 * 4096   # capped at the sliding window
print((global_layers + local_layers) / 8192)   # 0.75

A quarter off the dominant memory cost at long context is worth real money in batch size, and the saving is available at every context length above the window. Below 4,096 tokens the two layer types are identical in cost, because the window never binds.

A quarter sounds modest until you notice what the cache competes with. Weights are a fixed cost paid once per GPU; the KV cache is paid per concurrent request, and it is what decides how many conversations fit alongside the weights. Cutting it by a quarter raises the number of simultaneous long conversations a card can hold by roughly a third, and throughput on a batched server is very nearly linear in that number. This is the practical reason Gemma 2 27B is servable on hardware where a fully global model of the same shape is awkward.

There is a compute saving too, though it is the smaller of the two. Attention arithmetic in a local layer grows linearly with sequence length once the window binds, rather than quadratically, because each query attends to a fixed 4,096 keys instead of to everything before it. At 8,192 tokens that is a modest gain; the design pays off far more at Gemma 3’s context lengths, which is exactly where Google took it next.

Why it bounds effective reach

Here is the consequence that the memory story hides. Information can only travel further than 4,096 tokens by passing through a global layer. Local layers can move information forward in steps, since a token at position i attends to i - 4096 and that token attended to i - 8192 in an earlier layer, so multi-hop propagation exists. But direct, single-hop, long-range lookup only happens in the global layers, and there are half as many of those as there are layers.

That is why extending the window by editing a config does not work. You can set max_position_embeddings to 32,768 and the model will run, in the sense of producing tokens. What it will not do is reliably retrieve a fact from 25,000 tokens back, because the global layers are being asked to do long-range mixing over spans four times longer than any they were trained on, with RoPE frequencies that were fitted to 8,192. The failure is quiet: fluent output with the relevant detail dropped.

So the honest statement of Gemma 2’s long-context ability is that it has an 8,192-token window and roughly half its depth available for reaching across it. That is sufficient for chat and for single documents, and it is not a long-context model.

Where implementations get it wrong

• Uniform masking. A serving stack that applies the same causal mask to every layer, ignoring sliding_window, gives every layer global attention. The model still produces plausible text, which is exactly why this is hard to spot, but it is not the trained computation and quality drifts at long context.
• Cache eviction that is too eager. The mirror-image bug: applying the sliding window to all layers, including the global ones. Long-range recall collapses and nothing errors.
• Soft-capping. Gemma 2 also applies logit soft-capping in attention and on the final logits, and some fast attention kernels did not support it at release. Implementations that silently disabled soft-capping to use a faster kernel produced subtly different distributions. If a framework offers a flag for this, know which way it is set.
• Quantised repacks. A GGUF or similar conversion carries the attention pattern in its metadata. Older conversions made before the loader understood alternating attention exist in the wild and are not corrected by re-downloading a newer runtime alone.

How Gemma 3 pushed the same idea further

Gemma 3 kept the mechanism and changed the ratio: five local layers to each global one, with a smaller local window, and a much higher RoPE base frequency on the global layers so they can span the far longer context. The cache saving is proportionally larger, which is precisely what makes a 128K window plausible on a 4B model that people run locally.

The design lesson generalises past Gemma. When a small open model advertises a very long window, the interesting question is not the number but how many of its layers can see the whole of it. The version history records where each generation landed.

It also changes what a retrieval-augmented design should do with a Gemma model. If long-range mixing is concentrated in a minority of layers, the position of the relevant passage in the prompt matters more than it would on a fully global model: material the answer depends on is better placed near the question than buried thousands of tokens earlier. That is a cheap change to a prompt builder and it costs nothing to make, which is a better trade than most long-context tuning available at this size.