Tool Results Are Context Too

Every other block in the window was written by a person who saw its size. Tool results are generated at run time by systems that have no idea they are being billed per token, and they are the reason well-budgeted agents still run out of room.

The block nobody budgeted

A tool call produces a message that goes into the conversation array and is re-sent on every subsequent turn until something removes it. Nothing about that pipeline involves a size decision. The API returned what it returned, the SDK appended it, and the next request is larger by however many tokens that was.

The magnitudes are not marginal. A single SELECT * against a modest table, a directory listing of a node_modules, an un-paginated search API returning fifty results with full bodies, a verbose stack trace, an HTML page fetched without extraction — any of these can exceed the entire conversation that preceded it. And because the payload is machine-generated, it is dense in exactly the way that tokenizes badly: quoted keys, escaped strings, base64, UUIDs, deeply indented JSON. The same information as prose would cost a fraction.

There is a compounding effect too. A tool result that pushes the window near its limit makes the next tool call more likely to be truncated, so the model sees a partial result, so it calls the tool again with slightly different arguments, so the window grows again. Agents that appear to loop pointlessly are often doing exactly this, and a stopping condition catches the symptom without touching the cause.

Four payload shapes and what each needs

Generic truncation — take the first N characters — is the wrong treatment for three of the four common shapes, because it keeps the least informative part of each.

Whichever shape it is, the truncation must be visible to the model. A payload that silently ends looks complete, and the model will reason as though it were. A payload ending with [truncated: 412 of 5,180 rows shown] tells the model to paginate, narrow the query, or say the result was incomplete — which is the behaviour you wanted.

The adapter

Bounding belongs at the tool boundary, not in the model loop, so that every tool gets it and no individual tool implementation has to remember:

type ToolSpec = {
  name: string;
  budget: number;                 // max tokens this tool may put in the window
  shape: "list" | "record" | "document" | "diagnostic";
  project?: string[];             // fields to keep for "record"
};

async function callTool(spec: ToolSpec, args: unknown, store: BlobStore) {
  const raw = await tools[spec.name](args);
  const ref = await store.put(raw);          // full payload always kept

  let view = raw;
  if (spec.shape === "record" && spec.project) view = pick(raw, spec.project);
  if (spec.shape === "list")       view = { items: raw.items.slice(0, 20),
                                            total: raw.items.length };
  if (spec.shape === "diagnostic") view = dedupeLines(raw);

  let text = render(view);
  const n = tokens(text);
  if (n > spec.budget) {
    text = spec.shape === "document"
      ? headTail(text, spec.budget)          // keep both ends
      : head(text, spec.budget);
    text += "\n[truncated: " + n + " tokens produced, "
          + spec.budget + " shown. full result id=" + ref + "]";
  }
  return { text, ref, produced: n };
}

Three properties are doing the work. The budget is declared per tool, because a schema lookup and a web fetch do not deserve the same allowance. The full payload is always stored, so nothing is actually lost. And produced is returned to your telemetry, which turns “which tool is eating the window” from an investigation into a sorted list.

Return a handle, not the haystack

The stronger pattern, once the adapter exists, is to stop returning bulk at all. The tool writes its result somewhere addressable and returns a reference plus a summary; a second tool reads slices of that reference on demand.

query_db(sql)      -> { rows: 5180, columns: [...], id: "rs_91f",
                         preview: <first 5 rows> }
read_result(id, offset, limit) -> <that slice>
filter_result(id, expr)        -> { rows: 12, id: "rs_92a", preview: ... }

Three things improve at once. The window cost of a large result becomes constant rather than proportional to the result. The model is given an accurate count, so it can decide to filter rather than scroll. And the expensive operation is not repeated — a re-read is a store lookup, not a second database query. This is the same idea as externalising working memory to files, applied to the tool boundary rather than to the agent’s own notes.

Old results are the second problem

Bounding each result solves the spike. It does not solve accumulation: twenty bounded results at 1,500 tokens each is still 30,000 tokens of window occupied by output that was relevant three steps ago. Tool results have the steepest relevance decay of anything in the window, and they should be the first thing evicted.

• Collapse superseded results. If the same tool was called twice with the same arguments, only the latest is meaningful; replace the earlier with a one-line note that it happened.
• Age out by step distance. A result more than k steps old becomes its own summary line — [read config.yaml at step 4: 3 services, ports 8080/8081/9090] — with the handle retained so it can be re-read if needed.
• Never evict the result the current plan depends on. This is why eviction wants a pin list rather than pure recency; the file being edited or the schema being written against must survive regardless of age.
• Keep errors longer than successes. A failed tool call is the reason the agent is doing what it is doing now, and evicting it invites the agent to retry the thing that already failed.

Two habits keep this from becoming a permanent maintenance burden. Instrument first: record produced against budget for every call and look at the distribution rather than the mean. Tool output is a long-tailed quantity — most calls return a few hundred tokens and one in fifty returns forty thousand — so a mean tells you nothing and a p99 tells you which tool needs a projection or a pagination argument. The ranking that produces is usually short, and fixing the top two entries recovers most of the window.

Then push the fix upstream where you can. A budget applied at the adapter is a safety net; a tool whose signature takes limit, offset and a field list is a tool that cannot overflow in the first place, and it also gives the model a way to ask for exactly what it needs rather than receiving everything and discarding most of it. The adapter should be catching genuine surprises, not routinely cleaning up after tools that were designed without a token cost in mind.