Llama 3’s Context Window: From 8K to 128K Across Releases

“Llama 3” names at least four releases with two different context lengths between them. The 8K figure and the 128K figure are both correct, for different weights, and the version digit after the decimal point is what decides which.

The documented lengths

These are the context lengths Meta states on the model cards published with each release. The dates are the release dates.

Release          Date          Sizes                  Context
Llama 3          2024-04-18    8B, 70B                8,192
Llama 3.1        2024-07-23    8B, 70B, 405B        131,072
Llama 3.2        2024-09-25    1B, 3B               131,072
Llama 3.2 Vision 2024-09-25    11B, 90B             131,072
Llama 3.3        2024-12-06    70B                  131,072

Note that the marketing figure and the configured figure differ by a rounding convention. Meta and everyone else say “128K”; the number in config.json is 131072, which is 128 × 1024. The 8K figure is likewise 8192. Nothing is being approximated — the two are the same number written in different units.

Meta’s model cards for these releases are published in the meta-llama/llama-models repository, with a per-release copy alongside the weights on Hugging Face.

What changed at 3.1

The 8K to 128K step is a sixteen-fold increase that arrived three months after the original release, on weights with the same architecture and the same tokenizer. It was not a new model family. It was the same pretraining run extended: Meta applied RoPE scaling to the positional encoding and then continued pretraining on progressively longer sequences, so the model had actually seen inputs of that length rather than merely being permitted to receive them.

That distinction matters when you read a context figure anywhere. A context window is two claims stacked: the positional encoding can address N positions, and the model was trained on sequences near N. The first is a config change and costs nothing; the second is compute and is the reason 3.1 took three months. The mechanics of the first are in how RoPE scaling extended the window.

It is worth keeping a third claim separate from both: that the model uses the window well across its whole range. A declared length is the point at which the software rejects your request, not a guarantee of uniform recall. Retrieval accuracy over a long context generally falls off before the limit, and it falls off unevenly — material near the start and the end of a long prompt tends to be recovered more reliably than material in the middle. So “fits in the window” and “will be used” are different statements, and a design that depends on a fact buried at position 60,000 is depending on the weaker one. If long-context recall is load-bearing for you, it is a property to measure on your own documents rather than to infer from a model card.

The 405B model is the one case where the served figure most often departs from the configured one for reasons of sheer size rather than policy; the 405B context window covers what that costs to host.

Llama 3.2’s 1B and 3B models keep the 131,072 figure despite being far smaller, because they were derived from the 3.1 models by pruning and distillation rather than trained fresh. The vision models keep it too, though images consume context in a way that is easy to underestimate — see which 3.2 sizes take image input.

Where the number lives in the files

For a Hugging Face checkpoint the authoritative field is max_position_embeddings in config.json. A Llama 3.1 config carries it alongside the scaling block that makes it reachable:

{
  "architectures": ["LlamaForCausalLM"],
  "max_position_embeddings": 131072,
  "rope_theta": 500000.0,
  "rope_scaling": {
    "rope_type": "llama3",
    "factor": 8.0,
    "low_freq_factor": 1.0,
    "high_freq_factor": 4.0,
    "original_max_position_embeddings": 8192
  }
}

The original Llama 3 config has the same rope_theta of 500,000, max_position_embeddings of 8192, and no rope_scaling block at all. The presence or absence of that block is the quickest way to tell a 3.0 checkpoint from a 3.1 one when the directory name has been changed.

For a GGUF file the equivalent metadata key is llama.context_length, and llama.rope.freq_base holds the theta. Quantising a model does not change either — a 4-bit GGUF of a 3.1 checkpoint still declares 131,072.

Why your server reports something smaller

The configured length is what the weights can address. The served length is what your runtime allocated key-value cache for, and it is routinely much smaller, because 128K of KV cache is expensive memory that most deployments do not have spare.

• llama.cpp and Ollama default the context to a fraction of the declared maximum and require --ctx-size (or the num_ctx option) to raise it. A prompt longer than that is truncated or rejected by the server, not by the model.
• vLLM reads max_position_embeddings but will refuse to start if the resulting KV cache does not fit, and tells you to lower --max-model-len. The error names both numbers, which makes it the clearest of the three.
• Hosted resellers publish their own served length, which may be well under 131,072 for the same weights. Two providers of “Llama 3.1 70B” can legitimately advertise different context limits.

So the answer to “what is Llama 3’s context window” has two halves, and the half that produces a context-length error on a local model is almost always the served one.

What 128K costs in memory

The reason nobody serves the full window by default is worth working out rather than asserting, because the number is larger than most people expect and it is the number that decides your deployment.

Assumptions, all from the Llama 3.1 8B config: 32 transformer layers, 8 key-value heads (grouped-query attention, so fewer than the 32 query heads), head dimension 128, and a KV cache stored in bfloat16 at 2 bytes per value. Both a key and a value are cached per head per layer, hence the factor of two.

bytes per token = 2 (K and V)
                x 2 bytes    (bfloat16)
                x 32         (layers)
                x 8          (KV heads)
                x 128        (head dim)
                = 131,072 bytes = 128 KiB per token

full window     = 128 KiB x 131,072 tokens
                = 16 GiB  for ONE request at full context

Sixteen gigabytes, for a single conversation, on top of the roughly 16 GB the 8B weights themselves occupy in bfloat16. On an 80 GB card that leaves room for a handful of concurrent full-context requests, and the arithmetic is worse for the 70B model, which has more layers. This is why serving stacks default the context low and why hosted providers price long-context requests the way they do: the cost is not the tokens, it is the residency.

Two levers change the result. Grouped-query attention is already one of them — with 32 KV heads instead of 8 the figure would be four times larger, which is the main reason Llama 3 uses GQA at every size where Llama 2 used it only at 70B. The other is quantising the cache itself to 8 bits, which halves the figure at some cost in quality and is supported by most serving stacks as an explicit flag.

Checking the figure yourself

Do not trust a directory name. Read the config of the checkpoint in front of you:

# Hugging Face checkpoint
python -c "import json;c=json.load(open('config.json'));\
print(c['max_position_embeddings'], c.get('rope_scaling'))"

# GGUF file
llama-gguf-hash --verbose model.gguf | grep context_length

# What vLLM actually served
curl -s localhost:8000/v1/models | jq '.data[0].max_model_len'

The third command is the one worth wiring into a health check. It reports the number that will actually reject your request, which is the only one your application experiences.