Context Windows as an Architectural Constraint

Every other fixed-capacity resource in a system gets an explicit policy: what is admitted, what is rejected, what happens at the ceiling. Context windows usually get none of that, which is why they fail in production rather than in design review.

A capacity, not a limit

The instructive comparison is a memory-constrained embedded system. Nobody writing firmware for 64KB of RAM treats the limit as an occasional annoyance. It is the first fact of the design: data structures are sized for it, allocation is planned, and the behaviour when memory runs out is specified rather than discovered.

A context window is that constraint with two differences, and both cut the wrong way. It is metered — you pay per token used, not just per token available — and it is soft, in that exceeding it produces truncation or an error at request time rather than a compile failure. Softness is what makes it get ignored. The system works in development, works in staging, works for the first three weeks of production, and then a user has a long conversation.

Treating it as a designed capacity means answering three questions before writing the assembler: what is the smallest window this system must run on, what is the guaranteed-available budget after reservations, and what happens when a request would exceed it. All three have answers; most systems have not written them down.

Five design consequences

• Every tool return is bounded at the schema level. Not truncated at the call site as a safety net — designed to be bounded, with pagination in the signature. A tool that can return unbounded output is a tool that can end the session, and no amount of downstream budgeting fully compensates. If a tool can return a million rows, it must take a limit and return a handle; the handle pattern is the general form.
• State is a document, not a transcript. Anything the system must not forget lives in a structure with defined fields and update semantics, not in a message history that is being compacted behind it. This is the difference between a system that degrades gracefully at turn eighty and one that forgets the requirement.
• Loops have step budgets as well as token budgets. An agent that can take unbounded steps will eventually take enough of them to fill any window. The two limits are related but not interchangeable, and both belong in the design — stopping conditions are the mechanism.
• Long documents get an interface, not an inclusion. If any input can exceed the budget, the architecture needs a read interface for it — search, outline, page — rather than a plan to include it. Deciding this early is cheap; retrofitting it after everything assumes whole-document access is not.
• The prompt has a maximum size and it is asserted. A test that renders the largest realistic context and asserts it fits with margin catches the regression where someone adds three tool schemas, on the day it is introduced rather than three weeks later.

Decide the overflow behaviour first

This is the decision that most distinguishes a designed system from one that merely has an allocator. When the assembled context exceeds the budget, exactly one of four things should happen, chosen in advance, per block:

The one behaviour that must never be in the list is silent truncation. It is the default in a distressing number of framework code paths: the input is trimmed to fit, the request succeeds, and the model answers without material it was supposed to have. No error, no log line, no signal in the output — just a subtly wrong answer whose cause is unrecoverable after the fact. If your system can truncate, it should at minimum record what it dropped.

Different blocks deserve different behaviours, which is the point of deciding per block. Tool results: evict. History: compact. The safety policy: never — if it does not fit, refuse, because a request that cannot carry its own constraints is a request that should not be made.

Why a bigger window is not the fix

The standing temptation is to wait for larger windows. They keep arriving, and they genuinely help. They do not remove the constraint, for four reasons that are independent of each other.

Cost scales with what you use, not what is available. A million-token window that you fill is billed for a million tokens on every turn, and the quadratic growth of a long session applies at every scale. A bigger window changes when you hit the wall, not whether the bill grows with the square of the conversation.

Usable length lags advertised length. The gap between the two is documented, and the reasonable planning assumption is that the whole window is not equally usable regardless of what the spec sheet says.

More room means more contradictions. A larger window holds more superseded state, and the contradiction problem behind session decay is caused by accumulation, which a bigger window enables rather than solves.

Prefill has a latency cost. Reading 400,000 tokens takes measurable time before the first output token appears. For anything interactive, a full window is a slow window, and cache hits only help where the prefix is stable.

A larger window is a larger budget, and every rule here is about spending a budget well. The discipline does not become unnecessary; it becomes better funded.