RoPE Scaling in Llama 3.1: How the Context Window Was Extended Past 8K

Llama 3.1 went from 8,192 tokens to 131,072 with the same architecture and the same tokenizer. The config change that made it addressable is five lines long, and the most useful thing about it is that the obvious arithmetic on those five lines gives the wrong answer.

The four numbers

Every Llama 3.1, 3.2 and 3.3 checkpoint carries this block in config.json. The original Llama 3 has no such block.

"max_position_embeddings": 131072,
"rope_theta": 500000.0,
"rope_scaling": {
  "rope_type": "llama3",
  "factor": 8.0,
  "low_freq_factor": 1.0,
  "high_freq_factor": 4.0,
  "original_max_position_embeddings": 8192
}

• rope_type: "llama3" — names the scaling scheme. This is not linear interpolation and not NTK-aware scaling; it is a specific piecewise method, and the name is how a framework knows which code path to take.
• factor: 8.0 — the divisor applied to the low-frequency rotation rates only.
• low_freq_factor: 1.0 and high_freq_factor: 4.0 — the two thresholds that decide which frequencies are scaled, left alone, or smoothly interpolated between.
• original_max_position_embeddings: 8192 — the context length the model was originally trained at. It is the reference for the thresholds, not a limit.

Note also what did not change: rope_theta is 500,000 in both Llama 3 and Llama 3.1. A common assumption is that the theta was raised to extend the context. It was not; Llama 3 already used a large theta, and 3.1 extended the window through scaling plus training instead.

Why factor 8 is not 65,536

The tempting derivation is 8,192 × 8 = 65,536, and it is wrong twice over. The shipped context is 131,072, which is sixteen times the original, not eight. So the factor is plainly not a multiplier on the context length, and treating it as one leaves you unable to explain a two-fold discrepancy.

Two things are going on. First, the factor is applied per frequency band rather than to the sequence length: it divides the rotation rate of the low-frequency dimensions, leaves the high-frequency dimensions untouched, and interpolates in between. There is no single number it multiplies. Second, and more importantly, the config only makes long positions representable. What makes them usable is that Meta continued pretraining the model on progressively longer sequences, stepping up to 128K, so the model actually saw inputs at that length. The 131,072 figure is an outcome of that training programme, not a consequence of the numbers in the config.

This is the general shape of every context-extension claim you will read. A positional-encoding change is cheap and buys addressability; the expensive part is the training that makes the model competent at those positions. A config edited to declare a longer window without the training gives you a model that accepts long inputs and degrades across them, which is why community “extended context” re-uploads are worth treating sceptically.

What RoPE encodes

Rotary position embedding gives a token its position by rotating pairs of dimensions in the query and key vectors by an angle proportional to that token’s index. Each pair rotates at its own rate, set by theta, and the rates span many orders of magnitude — the fastest pairs complete a rotation in a handful of positions, the slowest take more positions than the model will ever see.

Because the attention score depends on the angle between two rotated vectors, and that difference depends only on the difference of their positions, RoPE encodes relative position. That is the property that makes it extensible at all.

The problem with a longer context is aliasing. A fast-rotating pair completes many full turns within the new window, so two tokens 3 apart and two tokens 3,003 apart can end up at nearly the same relative angle. The fast dimensions carry local detail — adjacency, word order — and they still work fine; the slow dimensions are the ones that were meant to distinguish far-apart positions, and at 131,072 positions they have not completed even one rotation, so they have no resolution left.

The band boundaries, derived

The llama3 scheme fixes this selectively rather than uniformly. Convert each rotation rate to a wavelength — the number of positions for one full turn, λ = 2π / freq — and compare it to two thresholds derived from the config. This arithmetic is exact, so here it is:

high_freq_wavelen = original_max_position_embeddings / high_freq_factor
                  = 8192 / 4
                  = 2048 positions

low_freq_wavelen  = original_max_position_embeddings / low_freq_factor
                  = 8192 / 1
                  = 8192 positions

For each frequency, with wavelength λ:

  λ < 2048            high frequency   unchanged
  λ > 8192            low frequency    freq / 8   (divided by factor)
  2048 <= λ <= 8192   in between       smooth interpolation

The interpolation weight is
  smooth = (8192 / λ - low_freq_factor)
         / (high_freq_factor - low_freq_factor)
         = (8192 / λ - 1) / 3

which runs from 0 at λ = 8192 to 1 at λ = 2048.

Read as intent: dimensions that complete a turn in under 2,048 positions are doing local work and are left exactly as they were, so short-range behaviour is preserved and the model does not get worse at the things it was already good at. Dimensions with wavelengths beyond the original 8,192-token window are the ones that would run out of resolution, so their rotation is slowed by a factor of 8, stretching them across the longer context. The band between gets a blend, so there is no discontinuity in the middle of the frequency spectrum.

That selective treatment is the whole idea, and it is why plain linear interpolation — which divides every frequency by the factor — measurably hurts short-context performance while this does not. It is also why the Llama 3.2 1B and 3B models, distilled from 3.1, carry the identical block: they inherit the positional scheme along with the weights.

The other rope_type values

llama3 is one entry in a family of schemes that a config can name, and knowing the others is useful because you will meet them in community re-uploads that claim extended context. They differ in exactly one respect: which frequencies they touch.

• default — no scaling. What the original Llama 3 uses, by having no rope_scaling block at all.
• linear — divide every frequency by the factor, uniformly. Simple, and it degrades short-range performance because the fast dimensions that were working fine get stretched along with the rest.
• dynamic — apply scaling only once the sequence exceeds the original length, interpolating by how far past it you are. Preserves short-context behaviour exactly, at the cost of the encoding changing partway through a long sequence.
• yarn — a per-band scheme in the same spirit as llama3, with additional attention-temperature correction. Used by several other model families, which is why you will see it in configs that are not Llama.
• llama3 — the piecewise scheme derived above: leave the fast dimensions alone, scale the slow ones, interpolate between.

The important thing about that list is that the scheme is a property of how the model was trained, not a knob you tune at inference. Loading a checkpoint with a different rope_type than it was trained under gives it positional signals it has never seen. This is the mechanism behind community “extended” variants that benchmark fine at short context and fall apart at long: the config declares a window the training never established.

Where this shows up in practice

• Old framework versions cannot load 3.1. rope_type: "llama3" arrived in Transformers 4.43. An earlier version raises an unrecognised-rope_scaling error, or — worse, in some third-party loaders — ignores the block and runs the model with unscaled positions, producing coherent output that degrades badly past 8K.
• Do not hand-edit the factor. Raising it does not extend the context; it changes the positional encoding away from the one the weights were trained against. Any real extension requires training.
• GGUF conversions must carry it. The rope parameters are metadata in the file. A conversion made by a tool that predates the llama3 scheme produces a model that works at short context and falls apart at long, with no error to point at.
• The block distinguishes 3.0 from 3.1. Its presence is the fastest reliable check on a checkpoint whose directory name you do not trust.
• Long context is not free at inference. KV cache grows linearly with sequence length and attention cost grows faster, so a declared 131,072 is only usable if you allocated for it, which is a separate matter from whether the encoding supports it. The memory arithmetic is worked through in Llama 3’s context window across releases, and it is the reason the served figure is usually lower — most visibly on the 405B model.