Budgeting the Context Window Across a Session

A context budget is not a guideline about keeping prompts short. It is a division of a fixed number of tokens between named claimants, made by code, before anything is rendered. Here is the code.

Reserve the output first

The window holds input and output together. Whatever room the answer needs is unavailable to the prompt, and this is the single most common arithmetic mistake in an assembler: filling to the window size and then discovering the model has nowhere to write. The mechanics of that — and the 400 it produces — belong to context window versus max output tokens. For the allocator, only the consequence matters:

available = window - reserved_output - safety_margin

The safety margin is not superstition. Your token count is an estimate made with a client-side tokenizer against a request the provider will re-serialise, and chat templates, tool schemas and role markers all add tokens you did not write. A margin of two to three percent of the window absorbs that; being wrong in the other direction costs a failed request at the end of an expensive assembly.

Reserved output is a property of the task, not of the model. A classifier that returns one word can reserve 64 tokens. A code generator that might emit a whole file cannot reserve less than a few thousand without occasionally truncating mid-function. Reasoning models make this sharper still, because the hidden thinking tokens come out of the same allowance — reserve for the thinking you cannot see, not just for the answer you can.

Floors, not percentages

The tempting design is percentages: 20% system, 30% history, 50% retrieval. It fails immediately, because the claimants are not elastic in the same way. The system block and the tool schemas have a size; they cannot be given 20% of anything. History and retrieved documents genuinely are elastic. So an allocator needs three quantities per block, not one:

• floor — below this the block is worthless and should be dropped entirely rather than shrunk. A retrieval block cut to 200 tokens is not a small retrieval block; it is one truncated document that will mislead.
• want — the size the block would use if nothing competed with it.
• priority — the order in which blocks are sacrificed when floors alone do not fit. Lower priority goes first.

The separation between shrink and drop is the part that carries most of the value. Truncating everything by a uniform fraction is the easy implementation and it degrades every block at once; a floor turns that into a decision to lose one block completely and keep the rest intact, which is almost always the better trade.

The allocator

Fixed blocks are paid first, floors are paid next in priority order, and whatever is left over is shared among the elastic blocks in proportion to what they asked for. Nothing exotic — but written down, testable, and the same on every request:

type Block = {
  id: string;
  fixed?: boolean;   // must be included whole or the request is invalid
  floor: number;     // drop below this rather than shrink
  want: number;      // size with no competition
  priority: number;  // higher survives longer
};

function allocate(blocks: Block[], available: number) {
  const out = new Map<string, number>();
  let left = available;

  // 1. Fixed blocks are not negotiable. If they do not fit, fail loudly.
  for (const b of blocks.filter(b => b.fixed)) {
    out.set(b.id, b.want);
    left -= b.want;
  }
  if (left < 0) throw new Error("fixed context exceeds window");

  // 2. Pay floors in priority order; anything unfunded is dropped.
  const elastic = blocks
    .filter(b => !b.fixed)
    .sort((a, b) => b.priority - a.priority);

  const funded: Block[] = [];
  for (const b of elastic) {
    if (left >= b.floor) { out.set(b.id, b.floor); left -= b.floor; funded.push(b); }
    else out.set(b.id, 0);      // dropped, not starved
  }

  // 3. Share the remainder in proportion to unmet demand.
  const demand = funded.reduce((s, b) => s + (b.want - b.floor), 0);
  if (demand > 0 && left > 0) {
    for (const b of funded) {
      const extra = Math.floor(left * (b.want - b.floor) / demand);
      out.set(b.id, Math.min(b.want, out.get(b.id)! + extra));
    }
  }
  return out;
}

Two properties are worth stating because they are what make it worth having. It is total — every block gets a number, including zero — so downstream code never has to guess whether a block was omitted deliberately. And it is pure, so the whole of your allocation policy is covered by tests that run in milliseconds and need no model.

A worked split

Assume — and every number here is an assumption, substituted for your own — a 128,000-token window, 4,000 reserved for output, and a 3,000-token safety margin. That leaves 121,000 available. Four claimants:

Fixed blocks take 8,200, leaving 112,800. Floors take 6,500, leaving 106,300 to share. Unmet demand is 56,000 for history and 77,500 for retrieval, 133,500 total — so history gets an extra 106,300 × 56,000 / 133,500 ≈ 44,600 and retrieval gets ≈ 61,700. Final split: history 48,600, retrieval 64,200. Both are under their want, so both will compact, and both know by how much before either one runs.

Now change one input and watch the policy earn its keep. Move to a 32,000-token window with the same reservations: available is 25,000, fixed takes 8,200, floors take 6,500, and only 10,300 is left to share. Nothing is dropped, but retrieval lands near 8,500 — about one and a half documents. That is the moment to notice that your retrieval step should be returning three documents rather than twelve, which is a filtering decision the allocator has just made visible.

Where a budget goes wrong

• Counting the wrong string. The budget must be computed over what is actually sent — after the chat template, with tool schemas serialised exactly as the SDK will serialise them. Counting the raw text under-reports, sometimes badly.
• An unbounded block that nobody declared. Tool results are the usual culprit: they arrive between requests, are appended by the framework, and never passed through allocate at all. Give them a block or they will quietly own the window.
• A budget that changes on every request. If the allocation shifts by a few tokens each turn, the prompt prefix changes each turn, and every cache hit is lost. Round allocations to a coarse granularity so the prefix is stable — the cache interaction is its own subject.
• Silent overflow. The failure mode of a missing budget is not an exception; it is a provider or a framework truncating from one end without telling you, which produces a model that has simply not read part of its instructions.