A Pre-Deploy Check That a Prompt Still Fits the Context Window

A prompt does not outgrow its context window on the day somebody edits it. It outgrows it on the day a retrieved document is longer than usual, three releases after the edit that used up the headroom. A pre-deploy check catches the edit, which is the only moment you can still act.

Write the budget down as arithmetic

The check is not “is the prompt smaller than the context window”. The context window holds input and output together, so the number you must stay under is smaller than the advertised one:

budget = context_window
        - max_tokens          # what you have reserved for the answer
        - thinking_budget     # if the model spends tokens reasoning
        - headroom            # growth you are willing to absorb before the next check

Every term is a decision, so put them in one file the test imports rather than in the test. When somebody raises max_tokens from 1,024 to 8,192 to fix a truncated answer, the gate should tighten by 7,168 tokens automatically and fail loudly, because that change has silently spent the same headroom the prompt was going to need.

Assemble the prompt production will send

The most common way this gate lies is that it measures the template rather than the request. The thing that goes to the provider is not your system prompt; it is the system prompt plus every tool schema, plus the few-shot examples the selector chose, plus the retrieved context, plus the conversation so far. Tool schemas in particular are easy to forget and are not small — Anthropic’s token counting documentation shows a single-tool request at 403 input tokens against 14 for the same message with no tools, which is the whole cost of one modest schema and its description.

So the check must call the same assembly function production calls, with a fixture that stands in for the largest realistic inputs. If you cannot call that function from a test, that is the finding: the assembly logic is entangled with the request path and nothing can measure it. Extracting it is the fix, and it pays for itself in template tests as well.

Count the whole request, not the parts. Summing a count for the system prompt, a count for the tools and a count for each message will overshoot, because every one of those sub-counts includes the fixed overhead the provider adds around a request, and it will also miss the structural tokens that delimit roles and content blocks. Send the assembled request to the counter once and use the number it returns.

Count with the model’s own tokenizer

Characters divided by four is a fine intuition and a bad gate. Two things break it. Tokenizers differ between model families, so a count that is comfortable on one provider is over the line on another; and tokenizers change within a family. Anthropic’s token counting page states plainly that Claude 4.7 and later models use a newer tokenizer under which the same input text produces approximately 30 percent more tokens than on earlier models, and tells you to recount against the model you plan to use rather than reusing counts from an earlier one. A gate carrying a number measured on the old tokenizer passes a prompt that no longer fits.

Anthropic exposes a counting endpoint that accepts the same structured request body as a completion — system prompt, tools, images, documents — and returns a single input_tokens field. The documentation records it as free to use, rate limited separately from message creation, and an estimate that may differ from the billed count by a small amount.

import anthropic

client = anthropic.Anthropic()

def count(model: str, request: dict) -> int:
    resp = client.messages.count_tokens(
        model=model,
        system=request["system"],
        tools=request["tools"],
        messages=request["messages"],
    )
    return resp.input_tokens

Because the endpoint is free and separately rate limited, running it in CI does not consume the budget the rest of your suite is competing for. Where a provider offers no such endpoint, use its published tokenizer library rather than an approximation, and mark in the test which model the count belongs to.

The gate

1. Put the budget terms in config/context-budget.json, keyed by model id: context window, reserved max_tokens, reasoning budget, headroom.
2. Add a fixture directory of worst-case inputs — the longest document your retriever can return, a conversation at your maximum retained turn count, the largest tool set any route registers.
3. Write one test per route that assembles the request from the fixture, counts it, and asserts the count is under the computed budget. Print the count and the budget in the failure message, and the delta.
4. Run it in the deploy pipeline, not only on pull requests, so that a change to a fixture or a config file that lands by another path still trips it.
5. Fail the deploy. A warning here is read once and then never, because the number creeps by a hundred tokens at a time and no single increment looks alarming.

Why the average prompt is the wrong input

Prompt size is not normally distributed around a typical case; it has a long right tail set by whatever the retriever found. The requests that overflow are the ones where a user pasted a large file, or the search returned ten long chunks instead of three short ones. A gate measured on a representative prompt is measuring the part of the distribution that was never going to fail.

The practical version of “worst case” is the cap your own code enforces. If the retriever is capped at eight chunks of 1,500 tokens, the fixture is eight chunks of 1,500 tokens, and if there is no cap then the gate cannot be written — which is itself worth discovering before a deploy rather than after. Pair this with a deliberate truncation strategy, because a budget you enforce by cutting text is only safe if the cutting is deterministic and cuts the part you meant.

One last thing the gate should print even when it passes: the headroom in tokens and as a percentage of the budget. A pass at 4 percent headroom and a pass at 40 percent are the same green tick and entirely different situations, and the trend across releases is what tells you whether the next model change has room to land. If that number is drifting down every release, the fix is not a larger budget — it is finding which part of the assembled prompt grew, which the same fixture makes easy to answer by counting each component in isolation for diagnosis, even though the gate itself counts the whole.