Llama 3.1 405B's Context Window: The Documented Number and the Real One

Meta’s Llama 3.1 model card gives 128K tokens for all three sizes, 405B included. The checkpoint’s own config file gives 131,072. Neither is the number your server will let you send, and the gap between them is the interesting part.

The documented number

Meta published Llama 3.1 in July 2024 in three sizes — 8B, 70B and 405B — and the model card states a context length of 128K tokens for every one of them. That card is maintained in the meta-llama/llama-models repository on GitHub, and it is the primary source for this figure; the copies on model hubs are downstream of it. The 405B is a dense model, not a mixture-of-experts one, so every parameter participates in every token — a fact that matters below, when the window has to be paid for.

The same card records a knowledge cutoff of December 2023 for the Llama 3.1 family, which is a different question from context length and is covered separately in the cutoff table. Context is what you can send; cutoff is what the weights already contain.

One ambiguity to resolve before quoting the figure to anyone: Meta published the 405B in more than one form — a base model, an instruction-tuned model, and an FP8-quantised build of the latter, made because 405B parameters at sixteen bits is an awkward amount of hardware. All of them document the same context length, so the number is not in dispute across variants. What differs is the memory footprint, and therefore how much a given deployment can afford to allocate to the window — which is the subject of most of this page.

128,000 or 131,072?

Open the checkpoint’s config.json and the field that governs position handling reads:

"max_position_embeddings": 131072,

131,072 is 128 × 1024. “128K” in the marketing sense is 128,000; 128K in the binary sense is 131,072. Meta rounds down to the decimal number in the card and the implementation uses the binary one, so the two are not in conflict — the card is quoting a floor. If you are writing a length check, use the smaller figure and leave the difference as headroom. Nothing good happens at the boundary anyway; see what happens when you exceed it.

How an 8K model got a 128K window

Llama 3, released three months earlier in April 2024, documented 8K. Llama 3.1 did not train from scratch at 128K — that would be prohibitively expensive, because attention cost grows with the square of the sequence length. Meta describes a continued-pretraining stage that extends the window in steps, paired with a change to how positions are encoded. The config.json records that change too:

"rope_scaling": {
  "rope_type": "llama3",
  "factor": 8.0,
  "low_freq_factor": 1.0,
  "high_freq_factor": 4.0,
  "original_max_position_embeddings": 8192
}

The original_max_position_embeddings of 8192 is the honest record of where the model started, and factor: 8.0 is the stretch applied to reach 65,536 — with the rest of the way to 131,072 coming from how the scheme treats different frequency bands differently. The mechanism has its own page in RoPE scaling. The practical consequence is that the window is extended rather than native, and quality across it is not flat: a model can address token 120,000 without attending to it as reliably as it attends to token 500.

What 128K costs in memory

The window you can actually use is set by the key-value cache, not by the config. Every token you send or generate leaves a key and a value vector in every layer, and those stay resident for the life of the request. The arithmetic is worth doing once, because it explains every “why can I only send 32K” question about this model.

From the published 405B config: 126 layers, 8 key-value heads (grouped-query attention, down from 128 attention heads), and a head dimension of 128. Per token, in bf16 (2 bytes per number), storing both a key and a value:

2 (K and V)
  x 126 layers
  x 8 kv_heads
  x 128 head_dim
  x 2 bytes            = 516,096 bytes ≈ 0.49 MB per token

0.49 MB x 128,000 tokens ≈ 63 GB   for ONE request at full context

Assumptions, stated: bf16 cache, no cache quantisation, no paged reuse, one sequence. Sixty-three gigabytes is on top of the weights, which for 405B parameters in bf16 are roughly 810 GB before any quantisation — which is why Meta also released an FP8-quantised 405B build, and why this model is served on multi-node GPU clusters rather than on a workstation. A deployment that advertises a shorter maximum is usually not disagreeing with Meta; it is reporting how much cache it budgeted per sequence.

Grouped-query attention is what keeps that figure merely large. With 128 key-value heads instead of 8 the same calculation lands near 1 TB per request, and a 128K window would not be servable at all.

128K addressable is not 128K usable

The context length is a statement about what the model will accept without erroring. It is not a statement that every position in the window is attended to equally well, and conflating the two is the source of a great deal of disappointment with long-context models in general.

Two mechanisms pull in that direction. The first is the extension itself: positions beyond 8,192 are reached by rescaling rotary embeddings rather than by having been trained at that length from the start, and the further out you go the further the model is from territory it saw often during pretraining. The second is a general property of long-context transformers, documented well before Llama 3.1 existed — Liu and colleagues published “Lost in the Middle” in 2023, showing that models recover information placed at the beginning and end of a long input more reliably than information placed in the middle, and that the effect grows with input length.

Neither finding is specific to the 405B and neither says the window is useless. What they say is that position matters, and that gives you three things to do with a long prompt:

• Put the instruction last. If a 100,000-token document is followed by the question, the question is in the strong recency position. If the question comes first and then the document, it is 100,000 tokens away from where the answer is being generated.
• Do not fill the window because it is there. Retrieving the six relevant sections of a manual generally beats pasting the manual, costs a fraction as much, and is faster — prefill time grows with prompt length whatever the quality does.
• Test at the length you will actually use. A retrieval prompt validated at 4K and deployed at 100K has not been validated. This is your own evaluation to run; no published figure answers it for your task.

Checking what your deployment allows

Never assume the documented number is the served number. Two checks take a minute each.

1. For a checkpoint on disk, read the field directly rather than trusting a README: python -c "import json;print(json.load(open('config.json'))['max_position_embeddings'])".
2. For an OpenAI-compatible server, send one over-long request and read the error. vLLM answers with the served maximum in the message itself, which makes it a reliable probe: it names both the limit and the number of tokens you asked for.
3. For a hosted API, look for a context field on the model listing and treat it as authoritative over the model card, because the host is the one enforcing it.

The last case is the one that catches people out. Two providers serving the same 405B weights can advertise different maxima, and both are telling the truth about their own deployment.