Context Window vs Max Output Tokens: Not the Same Number

Nearly every 400 from a chat completions endpoint is one of two mistakes, and they have opposite fixes. Confusing them is the most common self-inflicted outage in this whole subject area.

Two limits, not one

The context window is a shared budget. Everything the model attends to has to fit in it: system prompt, tool schemas, conversation history, the current user message, and the answer it is about to write. Input and output are not separately budgeted here — they compete for the same space.

The maximum output is an independent cap on how many tokens a single response may contain, set by the provider per model. It is very often much smaller than the context window: a family may advertise a 200,000-token context and cap a single completion at a small fraction of that. Having room in the window does not entitle you to a long answer.

And max_tokens (or max_completion_tokens) is your own request-level ceiling, which must respect both of the above. It is not a target, it is a reservation: on most APIs the space is reserved out of the window when the request is admitted, whether or not the model uses it.

The output cap looks arbitrary until you know where it comes from. A generating request holds a KV cache slot for its entire duration, and that slot grows with every token produced. A single request allowed to generate for a very long time occupies scarce accelerator memory and blocks a batch slot that could have served many short requests, so providers cap generation length to protect throughput. It is a scheduling constraint expressed as a limit, which is why it does not move when your prompt gets shorter.

And there is a third limit that gets confused with both, because it is also counted in tokens and also returns a 4xx: the rate limit. Tokens per minute is a property of your account and the model, not of the request, and it produces a 429 rather than a 400. If your error says rate_limit_exceeded and mentions a per-minute figure, no amount of prompt trimming will help within that minute — you need backoff, not truncation. Three limits, three different fixes, all denominated in the same unit.

Telling them apart from the error

The classic OpenAI-style message when the shared budget is exceeded has been stable for years and names all four numbers:

This model's maximum context length is 8192 tokens. However, you
requested 8500 tokens (8000 in the messages, 500 in the completion).
Please reduce the length of the messages or completion.

Read it carefully and it tells you which lever to pull: 8,000 + 500 exceeded 8,192, so you can either trim the prompt or lower max_tokens. Either works. Contrast with the output-cap error, which is a different shape entirely — Anthropic’s is an invalid_request_error along the lines of max_tokens: 32000 > 8192, which is the maximum allowed number of output tokens. Here trimming the prompt does nothing at all. The only fix is to ask for less output, because you have hit a per-model constant that has nothing to do with how full the window is.

The diagnostic is therefore: if the message mentions your prompt length, it is the window. If it mentions only max_tokens and a model constant, it is the cap.

The formula

One expression prevents both errors. Compute it on every request rather than hardcoding a number that was safe when you wrote it:

RESERVE = 64   # slack for template scaffolding you did not count

def safe_max_tokens(prompt_tokens, context_window, output_cap, want):
    room = context_window - prompt_tokens - RESERVE
    if room <= 0:
        raise PromptTooLong(prompt_tokens, context_window)
    return max(1, min(want, output_cap, room))

The RESERVE is not superstition. Your count of the prompt is almost never exactly the provider’s count, because chat templates and injected system content are added server-side — the subject of why your count differs from the bill. A few dozen tokens of slack converts a class of production 400s into nothing.

The failure that is not an error

The dangerous case does not raise. If the model reaches your max_tokens before it reaches a natural stop, you get a 200 OK with a truncated answer, and the only signal is a field: finish_reason: "length" on OpenAI-shaped APIs, stop_reason: "max_tokens" on Anthropic-shaped ones.

If you are parsing JSON out of that response, truncation means aJSONDecodeError at some random depth, and teams routinely spend an afternoon blaming the model’s formatting for what is a budget bug. Check the finish reason before you parse. Always. It costs one line and it is the difference between a clear log message and a mystery.

Where reasoning tokens land

On reasoning models the arithmetic gains a term that is invisible in the response body: the hidden thinking tokens are counted as output, and they are spent before the visible answer starts. Set max_tokens to 500 on a model that wants 2,000 tokens of reasoning and you get a response that is complete, well-formed, billed, and empty of content.

Budget those explicitly rather than discovering them — the reasoning tokens page works through the accounting.

The general shape to internalise is that all three numbers are consumed by things you did not write. Tool schemas, chat-template scaffolding and provider-side preamble all occupy the window; reasoning occupies the output cap; and a long tool result can exhaust the window several turns after the request that produced it. Treat the window as a resource with several claimants rather than as a limit on your own prose, and the arithmetic stops surprising you.