Auditing Context Window Assumptions Before a Model Migration

The prompt size that matters is not the one in your test fixture. It is the 99.9th percentile of what production actually sends, which is usually several times the median and is produced by inputs nobody designed for — the customer who pasted a whole log file, the conversation that ran for two hundred turns.

Why the advertised number is not the answer

A context window is a ceiling on input plus output measured in that model’s tokens. Three things make the advertised figure a poor basis for a migration decision.

First, tokenizers differ. The same text does not produce the same token count on two different model families, and the difference is not a small constant. Model families have changed tokenizer between generations within a single vendor, with the same text costing materially more tokens afterwards. So a prompt measured at 180,000 tokens on your current model is not 180,000 tokens on the target.

Second, the window is shared with the output and, on models that reason before answering, with the reasoning tokens. A window is not an input budget.

Third, the effective limit can be lower than the advertised one for your account or your deployment tier. The number in the marketing table is the model’s capability; what your key is permitted to send is a separate question, and one worth confirming with a probe request rather than assuming.

Four things share the window

Write the budget out explicitly before measuring anything, because most overflows are caused by a component nobody counted.

• The system prompt and tool definitions. Tool schemas are frequently the largest fixed cost in an agent and are almost never counted, because they do not look like prompt text. A dozen tools with detailed descriptions is a few thousand tokens on every single request.
• The conversation or retrieved context. The part that grows. In an agent loop, tool results accumulate here and are usually the fastest-growing component.
• The reserved output. Whatever cap you set has to fit. On some APIs the output cap is a required field and is validated against the remaining window at request time, so an oversized cap fails the request even when the input alone would have fit.
• Reasoning tokens, where applicable. On models that think before answering, that thinking is generated inside the same budget, is billed, and is often invisible in your logs. A response that truncates “for no reason” on a reasoning model is usually this.

Counting with the right tokenizer

Estimating with the wrong tokenizer is the single most common source of a wrong answer here, and the estimate is always confidently wrong rather than obviously wrong.

Anthropic exposes a token-counting endpoint, POST /v1/messages/count_tokens, which takes the same messages, system and tools you would send and returns input_tokens. Because it accepts the full request shape, it counts the tool schemas and the message envelope, not just your text — which is exactly what you need. Google’s API offers an equivalent countTokens call. OpenAI has no server-side counting endpoint; you count locally with tiktoken using the encoding for the specific model.

The asymmetry has a practical consequence. Counting for an OpenAI-shaped target is fast and free and can run in a tight loop over a million logged prompts. Counting for a target with a server-side endpoint costs a network round trip each, so you sample rather than enumerate — and you must count the whole request, since a local approximation of the message envelope is where the error creeps in.

Measuring your real distribution

1. Turn on logging of the input token count for every request, if it is not already on. Both APIs return it in the usage object on every response, so this is a logging change rather than an instrumentation project. Wait until you have at least a full week — weekly cycles are real, and Monday morning is not Saturday night.
2. Plot the distribution, not the mean. You want p50, p95, p99, p99.9 and the maximum. The maximum is the number that decides whether the migration breaks, and it will be far from the mean.
3. Take the 200 largest requests from the log and re-count each of them against the target model’s tokenizer. Compute the ratio of new count to old per request — and look at the spread, not just the average, because the ratio varies with content type. Code and non-English text usually shift more than English prose.
4. Add the fixed overheads at the target: the system prompt and the tool schemas re-counted with the same tokenizer, since those change too.
5. Add the output reservation. Use the largest cap any code path actually sets, not the typical one.
6. Compare the resulting total against the target’s window, and verify with one real request at that size. A probe that returns 200 settles the question in a way arithmetic cannot.

The re-count step is worth writing as a small script rather than doing by hand, because you will run it again for the next model:

import anthropic, json, statistics

client = anthropic.Anthropic()
ratios = []

for line in open("largest_200.jsonl"):
    rec = json.loads(line)
    new = client.messages.count_tokens(
        model=TARGET_MODEL,
        system=rec["system"],
        tools=rec.get("tools", []),
        messages=rec["messages"],
    ).input_tokens
    ratios.append(new / rec["old_input_tokens"])
    print(rec["id"], rec["old_input_tokens"], "->", new)

print("median ratio", statistics.median(ratios))
print("worst ratio ", max(ratios))

Reading the result

Three outcomes, and each has a different next step.

• Maximum well under the target window. Nothing to do. Record the headroom as a number and add an alert that fires when an observed request passes some fraction of it, so growth in prompt size is visible before it is fatal.
• Maximum near the window. The migration is viable but fragile. Find out what produces the largest requests — it is usually one code path, often an untrimmed conversation history or a retrieval step with no result cap. Bound that path before migrating, not after.
• Maximum over the window. A real constraint, and one with several possible answers depending on where the size comes from. Work through the options when the target window is smaller than the source before assuming the migration is off.

One habit makes all of this cheaper next time: store the counted input size alongside every request in your logs, and store which model produced it. A month of that data turns the next migration’s audit from a project into a query.