Qwen2.5's 1M-Token Context Variant

“Qwen2.5 supports one million tokens” is true of two specific checkpoints and false of every other model in the family. The difference is not a flag you turn on; it is a different repository, with a different config and an inference stack requirement attached.

Which models are the 1M ones

In January 2025 the Qwen team released two long-context checkpoints, announced on the Qwen blog and published on Hugging Face: Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M. The -1M suffix is the whole signal. A repository without it is a base Qwen2.5 instruct model, and those are documented at 131,072 tokens of context — a large window, but not this one.

Alongside the open weights, Alibaba Cloud offered a hosted qwen-turbo endpoint with a million-token window on Model Studio. That is a separate thing with separate pricing and separate limits; if you are calling an API rather than loading weights, the numbers on this page are not the ones that bill you. See the hosted Qwen output ceilings for how the input and output halves of a hosted window are quoted separately.

What the config actually says

Open config.json in either 1M repository and the long window is visible in two places. max_position_embeddings is set past a million rather than to 131,072, and there is a rope_scaling block that describes how the position encoding is stretched:

{
  "max_position_embeddings": 1010000,
  "rope_theta": 10000000.0,
  "rope_scaling": {
    "type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 262144
  }
}

Read that as a chain rather than as a single number. The model was trained to a long native window, and YaRN extrapolation multiplies it from there; the product is the figure in max_position_embeddings. The consequence people miss is that this is a property of the loaded configuration, not of the weights alone — an inference stack that ignores the rope_scaling block, or a serving flag that caps the sequence length lower, gives you a shorter window from exactly the same files with no error.

This is the single most common way the 1M window fails to appear. Under vLLM, --max-model-len is what the server enforces; set it below the config value and the config value is irrelevant. Under transformers, a long sequence will run and will simply exhaust memory rather than refuse.

It is worth confirming what the server actually decided rather than what you asked for. An OpenAI-compatible server will refuse an over-long request with an error naming the limit it enforced, and that error is the cheapest way to discover the real number:

This model's maximum context length is 262144 tokens.
However, you requested 300100 tokens (300000 in the messages,
100 in the completion). Please reduce the length.

If that message names a figure lower than the one on the model card, the gap is your serving configuration, not the checkpoint. The usual causes are an explicit --max-model-len, a KV-cache allocation too small for the requested length so the server clamps down at startup, or a stack version that does not implement the rope_scaling type in the config and silently falls back.

The attention changes underneath

Attention cost grows with the square of sequence length, so a million tokens is not 8× the work of 128,000 — it is closer to 64× if nothing changes. Two techniques described in the Qwen2.5-1M release make it tractable.

The first is Dual Chunk Attention, which reindexes positions so that distances the model never saw during training are mapped back into the range it did see. This is what lets a model trained on shorter sequences remain coherent far beyond them, rather than degrading into noise at the boundary. The second is sparse attention at inference time: rather than every query attending to every key, a selection step narrows the candidate set, which turns the quadratic term into something much closer to linear over the ranges that matter.

Both live in the inference stack, not in the checkpoint. The Qwen team published a modified vLLM fork alongside the models for this reason, documented in the QwenLM/Qwen2.5-1M repository. Running the 1M weights on an unmodified stack is supported and will produce output; it just gives up the sparse-attention speedup and puts you back on the quadratic curve.

What it costs to serve

The weights are the small part of this bill. A 7B model at BF16 is about 15 GB. The KV cache for a million tokens is far larger than that, and it is what decides whether the run is possible: cache size scales with sequence length, layer count, and the number of key-value heads, so it is fixed by the architecture and the length, and no amount of quantising the weights makes it go away.

Time is the other cost. Prefilling a million-token prompt is a large amount of arithmetic that must complete before the first output token appears, so time-to-first-token at full context is measured in tens of seconds rather than in hundreds of milliseconds, whatever the tokens-per-second rate afterwards looks like.

Two features of a modern serving stack change that arithmetic materially and are worth enabling before concluding the model is too slow. Chunked prefill splits a very long prompt into pieces that are interleaved with ongoing decoding, which stops one enormous request stalling every other request on the server — it does not make your own request faster, but it makes the deployment usable by more than one person. Prefix caching stores the computed keys and values for a prompt prefix so that a second request sharing it skips that prefill entirely. For the archetypal long-context workload — one large document, many questions about it — prefix caching is the difference between paying the prefill once and paying it per question.

What to expect at a million tokens

• The window is an input budget, not an output one. The generation ceiling is unchanged by the long-context variant. Filling the context does not let the model write a longer answer.
• Retrieval and reasoning degrade differently. Finding one fact placed somewhere in a very long document holds up far better than reasoning that has to combine many facts spread across it. A passing needle-in-a-haystack result is not evidence that aggregation over the same document works.
• Cost scales with what you send, every turn. There is no state between calls, so a million-token context resent on each turn of a conversation is a million tokens of prefill each time unless the stack caches the prefix.
• Check the tokenizer, not the character count. A million tokens is not a million words, and the ratio depends heavily on language and on whether the content is code. The Qwen tokenizer is the thing that decides whether your document fits.