RoPE Scaling in llama.cpp for Extending Context

Setting -c above what the model was trained for does not extend its context; it just runs the model at positions it has never seen. RoPE scaling is the transformation that makes those positions look familiar, and it is the flag most likely to leave you with output that is fluent, plausible and wrong.

What RoPE encodes, and why it runs out

Rotary position embedding does not add a position vector to the token. It rotates the query and key vectors by an angle proportional to position, in each of many two-dimensional slices, at a different frequency per slice. Because a dot product between two rotated vectors depends only on the difference of the angles, attention between two tokens ends up depending on the distance between them rather than on their absolute positions. That is the property that makes RoPE work at all.

The frequencies come from a base — rope_freq_base, often 10000 in older models and much larger in long-context ones — with slice i rotating at base^(-2i/d). High-frequency slices distinguish adjacent tokens; low-frequency slices carry long-range order. Train on 8192 positions and the model has seen the slowest slices complete only a fraction of a rotation. Ask for position 40000 and those slices are in angular territory that never appeared in training, and the model has no learned behaviour for it. The failure is not a crash: attention scores simply stop meaning what they meant.

Three scaling types, and the one you rarely pick

--rope-scaling takes none, linear or yarn, and llama.cpp documents the default as linear unless the model specifies otherwise. Modern GGUFs carry their scaling configuration in metadata, so the common case is that you set nothing and the correct method is applied for you.

• none — positions are used as-is. Correct when the model was actually trained to the length you are asking for, and the right choice when a GGUF has stale metadata that is scaling something that does not need it.
• linear (position interpolation) — divide every position by a factor, so position 16000 is presented as 8000. Simple, and it degrades the model’s ability to distinguish nearby tokens because it compresses the high-frequency slices along with everything else.
• yarn — interpolate the low-frequency slices, leave the high-frequency ones alone, and blend in between. This is why YaRN holds up better at large factors: local resolution is what linear scaling spends first, and YaRN does not spend it.

There is also the NTK-aware approach, which is not a separate value of --rope-scaling but a consequence of raising --rope-freq-base: increasing the base stretches every frequency at once, extending reach without an explicit interpolation factor. Several long-context releases were produced exactly this way, with a base of 500000 or higher baked into the checkpoint — the Llama 3.1 family is the widely-copied example, and its scaling configuration is worth reading before you override anything on a derivative of it.

The reciprocal trap in the two scale flags

llama.cpp exposes the same quantity twice, in opposite directions, and this is the single most common way to get RoPE wrong:

• --rope-scale N — “expands context by a factor of N”. Internally it stores 1/N.
• --rope-freq-scale N — “expands context by a factor of 1/N”. Stored as given.

To double the context you want --rope-scale 2 or --rope-freq-scale 0.5. They are the same setting. Passing --rope-freq-scale 2 when you meant to double does the reverse: it compresses positions into half the range, which is not an error and does not warn, and produces a model that behaves oddly at every length rather than only at long ones.

--rope-freq-base is separate again and takes an absolute frequency, defaulting to whatever the model carries. Set it and you override the checkpoint’s own value, so it is worth reading the loader output — the base is printed with the other hyperparameters — before deciding the model needs help.

The YaRN parameters and what they separate

YaRN has four tuning knobs plus an original-context field, all of which default to -1 or 0 meaning “take it from the model”:

• --yarn-orig-ctx — the context the model was trained at. 0 means read it from the checkpoint. Set this wrong and every other YaRN number is computed against the wrong baseline.
• --yarn-ext-factor — how much of the extrapolation mix to apply. 0 disables YaRN’s ramp entirely and gives you pure interpolation.
• --yarn-attn-factor — a magnitude scale on attention, which compensates for the fact that interpolating positions changes the typical size of attention logits.
• --yarn-beta-fast and --yarn-beta-slow — the two correction dimensions that decide where the ramp between “leave alone” and “interpolate fully” sits.

The reason to know they exist is not to tune them. It is that a GGUF converted from a checkpoint whose YaRN configuration did not survive conversion will silently use defaults, and the symptom is a model that is fine to its original length and degrades after it.

What wrong settings look like from the outside

There is no error message for bad RoPE settings. The model loads, it generates, and the damage shows up as behaviour:

• Local incoherence at every length. Over-compressed high-frequency slices blur adjacent positions, so word order and short-range agreement suffer even in a 200-token prompt. This is the signature of an unnecessary or reversed scale factor.
• Fine early, drifting late. Output holds together for the first few thousand tokens and loses the thread beyond the original training length. This is the signature of no scaling where scaling was needed.
• Retrieval failure without fluency failure. The model writes well and cannot find a fact you placed 30k tokens back. Long-range slices are the ones carrying that dependency, and they are the first casualty of a large linear factor.

A procedure that does not require guessing

1. Read n_ctx_train from the model load output. If the context you want is at or below it, set -c and stop — no scaling flags at all, and budget the cache from the --ctx-size formula.
2. Check whether the checkpoint already carries a scaling configuration. Modern long-context GGUFs do, and overriding it by hand is the most likely way to make things worse.
3. If you must scale, prefer --rope-scaling yarn with an explicit --yarn-orig-ctx set to the trained length, and let the other YaRN parameters default.
4. Express the factor once, with --rope-scale, and never mix it with --rope-freq-scale in the same command line.
5. Measure rather than eyeball. Run llama-perplexity over a long document at your target context with scaling off and on; a scaling setting that is helping produces lower perplexity at long context, and one that is hurting shows up as a worse number at short context too. See the perplexity tool.