Qwen2.5's Context Window and the YaRN Extension

Qwen2.5’s model cards quote 131,072 tokens of context for the mid-size and large checkpoints. The config.json in the same repository says 32,768. Both are correct, and the gap between them is a configuration change you have to make deliberately.

The documented numbers

Alibaba publishes context and generation lengths per size on each Qwen2.5 model card. As documented on the Hugging Face cards for the instruct checkpoints:

• Qwen2.5-0.5B, 1.5B and 3B (Instruct) — 32,768 tokens of context, 8,192 tokens of generation.
• Qwen2.5-7B, 14B, 32B and 72B (Instruct) — 131,072 tokens of context, 8,192 tokens of generation, where the figure above 32,768 requires the YaRN extension described below.

Two things about that list are worth pinning down before you build anything on it. The first is that the generation figure is separate and much smaller: 8,192 tokens is the documented output cap regardless of how large the context is, so a 128K context does not buy you a 128K answer. The second is that the small checkpoints do not have a longer mode at all — 32,768 is the whole story for 0.5B through 3B, and there is no scaling configuration that makes them behave like the 7B. See the separate output cap for what happens when generation runs into its own limit.

The primary source is the model card itself. For the 7B instruct checkpoint that is Qwen/Qwen2.5-7B-Instruct on Hugging Face, published by the Qwen team at Alibaba Cloud, whose model summary block carries the context and generation lengths directly. The wider family is described in the Qwen2.5 technical report (arXiv:2412.15115, December 2024).

Why the model card and config.json disagree

Open config.json in the 7B instruct repository and you will find "max_position_embeddings": 32768. That is not a mistake in the card. The two numbers measure different things.

max_position_embeddings is the length the model was trained to handle natively — the range of rotary position frequencies it has actually seen during pretraining and post-training. Beyond that point, the position encoding is being asked to represent distances the model has no experience of, and quality falls off sharply rather than gracefully. The 131,072 figure is what the model reaches when a length-extrapolation technique is applied on top, and the technique Qwen2.5 documents is YaRN.

Practically, this means the default behaviour of every runtime that reads config.json — Transformers, vLLM, SGLang, llama.cpp — is a 32,768-token window. If you send 60,000 tokens to an unmodified Qwen2.5-7B-Instruct deployment, you get a context-length error, not a silently degraded answer. That error is a feature; the alternative is worse.

What YaRN does, and the config that turns it on

YaRN (“Yet another RoPE extensioN”) is a method for rescaling rotary position embeddings so a model trained at one length works at a longer one. It was introduced by Peng, Quesnelle, Fan and Shippole in arXiv:2309.00071 (August 2023). The mechanism is worth one paragraph because it explains the trade-off that follows.

Rotary embeddings encode position as a rotation whose angle depends on a set of frequencies. Naive interpolation divides every frequency by the extension factor, which preserves long-range structure but crushes the high-frequency components the model uses to tell adjacent tokens apart. YaRN interpolates the low-frequency components, leaves the high-frequency ones alone, blends across the middle, and adds a small temperature correction to the attention softmax to compensate for the longer sequence. The result keeps local resolution while extending reach.

Qwen2.5 documents enabling it by adding a rope_scaling block to config.json:

{
  ...,
  "max_position_embeddings": 32768,
  "rope_scaling": {
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 32768
  }
}

A factor of 4.0 against an original length of 32,768 is where 131,072 comes from: 32,768 × 4 = 131,072. The arithmetic is not incidental — if you need 65,536 tokens rather than 131,072, set the factor to 2.0 rather than 4.0, and you will pay less of the cost described below. Older Transformers releases spell the key "type" rather than "rope_type"; both appear in circulation, and a config with the wrong spelling for your version is ignored rather than rejected, which is the failure mode worth watching for.

Enabling it in vLLM and SGLang

You do not have to edit the downloaded config. Both major serving frameworks accept the same block as a launch argument, which is preferable because it keeps the checkpoint pristine and the setting visible in your deployment manifest:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 131072 \
  --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}'

Two arguments, and both are required. --rope-scaling changes how positions are encoded; --max-model-len changes the length the server is willing to accept. Setting only the second gets you a server that accepts 131,072 tokens and produces deteriorating output past 32,768, which is the single most common way this is got wrong. Setting only the first gets you correct scaling and a server that still rejects the request.

There is a memory consequence to --max-model-len that is easy to miss on a first deployment. The KV cache is sized against the maximum sequence length, so raising it from 32,768 to 131,072 quadruples the per-sequence cache requirement and therefore cuts how many sequences fit concurrently on the same GPU. On a card that was comfortably serving a batch, the symptom of the change is a collapse in throughput rather than an error.

Why it is off by default

The obvious question is why Alibaba does not simply ship the config with YaRN enabled. The answer is in how the implementations apply it. In Transformers and vLLM, the YaRN configuration is static: the rescaling is applied to every request, not only to the long ones. A 1,000-token prompt sent to a server configured for 131,072 is being processed with position frequencies stretched for a sequence 130 times its length, and the model’s performance on short inputs degrades as a result.

Qwen’s own documentation makes exactly this recommendation: enable the extension only when you need it. The practical shape that takes in production is two deployments — a default one at 32,768 that takes the overwhelming majority of traffic, and a long-context one with YaRN enabled that takes the requests that need it, with the router choosing on measured input length. That is more machinery than a single server, but it is the only arrangement in which both classes of request get the model at its best.