Gemma 2's Context Window, and Why It Is Shorter Than Its Peers

Every Gemma 2 checkpoint has a context window of 8,192 tokens. Not 8,192 for the small one and more for the large one: the same figure at 2B, 9B and 27B. That looks miserly next to the 128K windows shipping around it, and the reason is not that Google ran out of budget. It is the same design decision that made Gemma 2 cheap to serve.

The number, and where it is written down

The 8,192-token figure is on the Gemma 2 model cards Google publishes alongside the weights, and it is also inside every checkpoint. You do not have to trust a summary of it, including this one. The field is max_position_embeddings in config.json:

# Reads the config only. Does not download the weights.
python - <<'PY'
from transformers import AutoConfig
cfg = AutoConfig.from_pretrained("google/gemma-2-9b-it")
print(cfg.max_position_embeddings)   # 8192
print(cfg.sliding_window)            # 4096
PY

Two numbers come back and both matter. The first is the window. The second is the size of the local attention span used by half the layers, which is the subject of the sliding-window page and the reason the first number is what it is. Google publishes the same values on the gemma-2-9b-it model card and in the Gemma documentation at ai.google.dev/gemma/docs.

It is the same at 2B, 9B and 27B

This is worth stating plainly because the usual mental model is wrong here. With most families, buying the larger model buys you more room: more parameters, more context, more of everything. Gemma 2 does not work that way. The 2B released in July 2024 and the 27B released in June 2024 have identical position-embedding budgets. Moving up a size buys capability per token, not tokens.

The practical consequence is that a prompt that overflows on Gemma 2 2B will overflow on Gemma 2 27B in exactly the same place. If you are hitting the ceiling, scaling up the model is not a fix; the fix is fewer tokens, a summarisation step, or a different generation of the family.

It also means a capacity plan built on the 27B transfers to the 2B without recalculating prompt budgets, which is genuinely convenient when you run a small model as a first-pass filter in front of a larger one. The two see the same amount of the conversation, so a router that sends short requests to the 2B and long ones to the 27B is routing on difficulty rather than on length, and the length test tells you nothing useful.

Why it stops at 8,192

Attention cost grows with the square of the sequence length, and the KV cache that makes generation fast grows linearly with it, per layer. Gemma 2 attacks the second of those directly: it alternates local attention layers, which see only a 4,096-token sliding window, with global layers that see the whole sequence. Roughly half the layers therefore never need to keep a cache longer than the local window.

That halves the memory a long conversation costs, which is what lets a 27B model serve on hardware that would otherwise be marginal. But it also sets the shape of the model’s long-range behaviour: only the global layers carry information across the full sequence, and there are half as many of them as there are layers. Pushing the window out to 32K or 128K on that architecture would mean the global layers doing the entire job of long-range mixing over a span four to sixteen times longer than they were trained on. Gemma 2 was trained at 8,192 and the window is not an artificial cap you can lift by editing a config; the RoPE frequencies and the training distribution both stop there.

So the trade is legible: Gemma 2 spent its long-context budget on being cheap to run rather than on being able to read a long document. For a model designed to be downloaded and run on one GPU, that is a defensible allocation, and it is the same allocation that shows up in other small open-weight families.

There is a second reason the number is 8,192 and not something larger, and it is about training rather than serving. Long-context ability is not free at training time either: it requires a long-context extension phase on documents that are genuinely long and genuinely coherent, and that data is scarcer and more expensive to assemble than the general pretraining mixture. A model that advertises a window it was never extended on has a number and not an ability. Gemma 2 declined to advertise one, which is a less flattering headline and a more honest artefact.

What actually fits in 8,192 tokens

The window is shared by everything: the system instruction you folded into the first user turn, all prior turns, the current prompt, and the tokens the model is about to generate. It is not 8,192 in plus whatever out.

• English prose runs somewhere around 0.75 words per token with Gemma’s tokenizer, so 8,192 tokens is very roughly six thousand words of input and output combined. Treat that as an estimate for sizing, not a budget to plan against.
• Code and structured text tokenize worse. JSON with long keys, deeply indented source, and anything with unusual identifiers all inflate.
• Non-English text inflates further, though less than it does on models with smaller vocabularies. Gemma’s 256K-entry vocabulary is unusually large for a model this size, and that is part of why.
• Reserve space for output explicitly. If you want a 1,000-token answer you have 7,192 tokens for everything else, and the failure mode when you get this wrong is a truncated answer rather than an error.

Count rather than estimate when it matters. The tokenizer is local and free to run, so there is no reason to guess:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
n = len(tok.apply_chat_template(messages, tokenize=True, add_generation_prompt=True))
print(n, "of 8192")

Counting the rendered chat template rather than the raw strings is the part that catches people out. The template adds four special tokens and two newlines per turn, so a fifty-turn conversation carries a few hundred tokens of pure formatting before any of your content is counted. On an 8,192-token budget that is not noise.

What happens when you exceed the window depends entirely on the layer you exceed it at, and the failure is not always an error. Hugging Face transformers will happily run a sequence longer than max_position_embeddings and produce degraded output rather than raising, because the position indices simply go past anything the model saw in training. A serving framework usually does raise, with a message naming both numbers, and llama.cpp-derived stacks tend to truncate from the front of the prompt instead. Front-truncation is the most dangerous of the three: it silently deletes your instruction, which on Gemma lives at the very start of the first user turn because the family has no system role to put it in. Know which behaviour your stack has before you rely on it.

What changed in Gemma 3

Gemma 3, released in March 2025, moved to a 128K-token window on the 4B, 12B and 27B sizes and 32K on the 1B, using a more aggressive version of the same idea: five local layers to each global one, a smaller local window, and a much higher RoPE base frequency on the global layers. If the 8,192-token ceiling is your binding constraint, the answer is a generation change rather than a size change. The version-history page lays out what else moved at the same time.