Testing Behaviour at the Exact Edge of a Model's Context Window

The documented context window is a number about the request, not about your text. Testing the edge means finding where the boundary actually falls for your request shape, and then proving the failure is one your application handles.

Why you cannot just count tokens

Tokenise your prompt, get 127,900 tokens, decide you are safely inside a 128,000 window, and the request still fails. Several things sit between your text and the count the provider computes:

• Message envelope overhead. Each message carries role markers and separators that are part of the serialised conversation and are counted. A conversation of 200 short messages pays this 200 times.
• Tool schemas. Function definitions are serialised into the request and counted as input. A dozen tools with descriptive parameter documentation is easily thousands of tokens that appear in no prompt you wrote — the subject of what tool schemas cost in tokens.
• The reserved completion. On most APIs the window is shared between input and output, so max_tokens is subtracted up front. OpenAI’s overflow error is explicit about this, reporting for example that you requested 4,927 tokens made up of 3,927 in the messages and 1,000 in the completion.
• Images and non-text parts. These convert to token counts by a provider-specific rule your text tokeniser knows nothing about.

So a local tokeniser gives you an estimate of one component. It is useful as a guard rail and useless as a boundary.

Finding the boundary empirically

The provider tells you the number. OpenAI returns an invalid_request_error with code context_length_exceeded and a message of the form “This model’s maximum context length is 128000 tokens. However, your messages resulted in 249114 tokens”. That second figure is ground truth for your exact request shape, tools included, and it is what you binary-search against. Anthropic exposes a token-counting endpoint on the messages API for the same purpose, which lets you ask before you send.

1. Build filler from a single token you have verified encodes to exactly one token, repeated. Lorem ipsum tokenises unevenly and makes the search converge on an answer you cannot reproduce.
2. Send the request with your real system prompt, your real tool schemas and your real max_tokens, varying only the filler length. Changing any of the others moves the boundary.
3. Binary search on filler length between a value that succeeds and one that fails, comparing the reported token count each time. Twenty iterations locates it precisely; five gets you close enough to write the test.
4. Record the resulting overhead — window minus filler minus max_tokens — as a constant with a comment naming the model, the tool set and the date. That constant is what your pre-flight check uses.

The three inputs

With the boundary located, the test is three cases and they assert different things:

import pytest

WINDOW = 128_000
OVERHEAD = 1_842        # measured for gpt-class model + our 9 tool schemas
MAX_TOKENS = 1_000
FITS = WINDOW - OVERHEAD - MAX_TOKENS

@pytest.mark.parametrize(
    "filler_tokens,expect",
    [
        (FITS - 100, "ok"),      # comfortably inside
        (FITS,       "ok"),      # exactly at the edge
        (FITS + 100, "overflow") # just over
    ],
    ids=["under", "at-edge", "over"],
)
def test_context_window_edge(filler_tokens, expect, app):
    prompt = filler(filler_tokens)

    if expect == "ok":
        result = app.summarise(prompt)
        assert result.text                      # not empty
        assert result.finish_reason == "stop"   # not "length"
    else:
        with pytest.raises(app.ContextTooLong) as err:
            app.summarise(prompt)
        assert err.value.limit == WINDOW
        assert err.value.used > WINDOW

The at-edge case is the one that finds bugs, and it is the one most suites skip because it feels redundant. It is not: an off-by-one in your own reservation arithmetic only shows up when the margin is zero.

What the failure has to be

The row this page comes from asks whether the failure mode is a clean error rather than silent truncation, and that is precisely the assertion. Four wrong behaviours are common, and each has its own check:

• Silent truncation. Some middleware drops the middle of the conversation to make it fit and returns a confident answer based on half the context. Detect it by planting a distinctive fact in the part that would be dropped and asserting the answer either uses it or the request fails — never that it succeeds without it.
• An empty completion. When the window is full there is no room to generate, and some configurations return a successful response with zero content and finish_reason of length. Assert on the finish reason, not on truthiness of the text; see handling an empty completion.
• A generic 500 to the user. The overflow is a client error you can act on. Assert your layer maps it to a typed exception carrying the limit and the used count, so the interface can say the conversation is too long and offer to start a new one.
• A truncation that is not deterministic. If you do truncate deliberately, the same input must produce the same truncation every time, or a cached answer and a live answer diverge (deterministic prompt truncation).

The 400 that gets retried four times

An overflow is a 400. It is deterministic: the same request will overflow every time, forever. Yet retry layers configured to retry on “any error” will send it three more times, adding latency to a failure that was already certain and, on providers that bill rejected requests, paying for it. Assert the call count:

def test_overflow_is_not_retried(app, transport):
    with pytest.raises(app.ContextTooLong):
        app.summarise(filler(FITS + 100))
    assert transport.call_count == 1

The same test is worth running for 401 and 422. The general rule is that only 408, 409, 429 and 5xx are retryable, and a suite that pins that rule catches the day someone widens the retry predicate to make an unrelated flake go away.