Effective Context Length vs Advertised Context Length
5 min read · updated August 3, 2026
“200k context” is a statement about what the serving stack will accept, not a promise about what the model can use. The two numbers are related but they are not the same, and the gap between them has been measured — by people who published their method.
What the advertised number is
For an open-weights model, the advertised figure is usually max_position_embeddings from the model config, possibly adjusted by a scaling factor, and it means: positions beyond this index are not defined, so the runtime will refuse. For a hosted model it is whatever the provider’s admission control allows, which also has to account for how much KV cache memory a request of that length occupies.
Neither definition contains any claim about accuracy. A model will happily accept 200,000 tokens and answer using only the last 3,000 of them, and nothing in the API will tell you it did. That is the entire problem: exceeding the window is an error you can catch, while exceeding the model’s useful range is a quality regression that looks like a normal response.
How windows get extended cheaply
It is worth knowing why the two numbers diverge so reliably, and the answer is that lengthening the advertised window is much cheaper than training the capability. With rotary position embeddings the position signal is a function you can rescale. Position interpolation (Chen et al., 2023) squeezes longer sequences into the range the model was trained on; NTK-aware scaling and YaRN (Peng et al., 2023) refine that by scaling different frequency bands differently.
These techniques work, and they need only a modest amount of fine-tuning at the longer length — often far less data than the model saw at its original length. So a 4k model becomes a 128k model by config change plus a short adaptation run. The number on the spec sheet moves by 32×. What the model has actually practised at 100,000 tokens moves by considerably less.
What the benchmarks measure
Three families of published evaluation exist and they answer different questions:
| Benchmark | Description |
|---|---|
| RULER | Hsieh et al., NVIDIA, 2024. Synthetic tasks — multi-needle retrieval, variable tracing, aggregation — generated at configurable lengths, scored against a threshold defined by a short-context baseline. Its contribution is the term effective context length: the longest length at which a model still clears the baseline. The paper reported that most of the models it evaluated fell below their claimed lengths, several by a wide margin. |
| needle-in-a-haystack | Kamradt, 2023. One fact, controlled depth, single retrieval. Cheap, intuitive, and the weakest test in the set — passing it demonstrates the floor. |
| LongBench / ∞Bench | Real-task suites — long-document QA, summarisation, code completion over repositories — rather than synthetic probes. Closer to what you care about, harder to interpret because task difficulty and length are entangled. |
RULER is the one to read if you read one, because it makes the distinction this page is about into a defined, measurable quantity, and because its synthetic tasks scale to any length without needing new labelled data. Its results are also the reason to be sceptical of a model card that quotes a window without quoting a benchmark at that window.
Finding your own knee
Published effective lengths are a prior, not an answer, because they are measured on tasks that are not yours. The procedure to get a number you can design against:
- Build fifty graded examples of your real task. Not needles. Actual questions over actual documents, with an answer key and a grader — string match where possible, a rubric-scored model-as-judge where not.
- Hold the task constant and vary only the padding. The relevant content stays identical; you add unrelated documents to reach 8k, 32k, 128k and whatever the maximum is. Any other change confounds the result.
- Score at each length and find where it falls off. You are looking for the length at which accuracy drops meaningfully below the 8k baseline. That is your effective length for this task, on this model, with this prompt.
- Design your cap below the knee, not below the spec. If the knee is at 48k on a 200k model, your retrieval budget is 48k. The remaining 152k is available and not useful, which is a strange sentence and a real constraint.
- Re-run it on every model change. This is a
refreshproperty, not a fixed one. A new point release can move the knee in either direction and the release notes will not mention it.
One practical note: run the sweep with the passages at several positions too, or you will conflate a length effect with the position effect described on lost in the middle. They have different fixes — the first is a budget, the second is an ordering.
A second practical note, about serving rather than about models: the advertised length can differ between two providers hosting the same open-weights checkpoint. The config permits one number; the deployment sets another, based on how much KV cache memory it is willing to commit per request and on whether it applied a scaling technique at all. So “this model has a 128k window” is a claim about the weights, while what you can actually send is a claim about the endpoint. Check both before designing around either, and re-check after a routing change, because a failover to a second host of the same model can quietly hand you a shorter window.
The habit that follows from all of this is small: record the length you tested at next to the model in your configuration, and treat any change to either as requiring a re-run. It is the same discipline as pinning a dependency version. The failure it prevents is the one where a model upgrade advertised as strictly better quietly moves the knee downward and your longest, most valuable documents start producing confident nonsense.