Scratchpads and Externalised Working Memory

An agent that needs to remember something has two places to put it: inside the window, where it is re-sent and re-billed on every turn, or in a file, where it costs nothing until it is read. The choice has an arithmetic answer more often than people expect.

The window is not the only memory

Context is a strange kind of storage. It is read on every single turn whether or not it is needed, it is charged per read, its capacity is fixed, and writing to it is append-only with no update operation. Almost no other storage system in computing has that profile, and it is a bad fit for most of what an agent wants to remember.

A filesystem — or an object store, or a key-value table; the substrate is irrelevant — is the opposite on every axis. Unbounded, addressable, mutable in place, and free until read. Externalised working memory means moving the agent’s notes, plans and intermediate results out of the transcript and into that store, and giving the model tools to read and write it.

The pattern is the same one behind returning a handle instead of a payload, applied to the agent’s own output rather than to a tool’s. The agent writes a plan to plan.md and refers to it by name; the plan is not in the window, and updating it costs a write rather than another copy appended to the transcript.

The break-even

Let D be the size of the document in tokens, N the number of remaining turns in the session, q the fraction of turns on which the agent actually needs it, and s the size of the slice a read returns. Assume a per-turn overhead c for the tool call and its result envelope.

in-window : D * N                       (re-sent every turn)
in-file   : q * N * (s + c)             (read only when needed)

file wins when   D  >  q * (s + c)

As with the static-versus-retrieved derivation, N cancels: the number of remaining turns does not enter the decision, only the per-turn comparison does. Put illustrative numbers in — all assumptions — with a 20,000-token document, reads needed on a third of turns, a 2,000-token slice and 200 tokens of envelope: the file side is 0.33 × 2,200 ≈ 730 tokens per turn against 20,000. The file wins by roughly a factor of 27.

It does not always win. A 400-token checklist consulted on every turn: in-window costs 400, file costs 1.0 × (400 + 200) = 600. The window is cheaper, and it is also simpler and has no failure mode where the agent forgets to read the file. Small, always-needed state belongs in the window; large or intermittently-needed state belongs outside it. The crossover sits roughly where D approaches the size of a single read plus its overhead, which for most tool surfaces means a few hundred tokens.

One term is missing from the model and is worth naming: reads cost turns, not just tokens. Each read is a round trip, so a document consulted constantly through a file interface adds latency and steps that a window-resident document does not. Where a step budget is the binding constraint rather than a token budget, the arithmetic shifts towards keeping things resident.

Four properties the window does not have

When the arithmetic is close — and it often is — these usually decide it.

• It survives compaction. This is the big one. Anything in the transcript is a candidate for summarisation and eventual loss; a file is not in the transcript. An agent whose plan lives in a file still has its plan at turn eighty, when the turns that created it are long gone. Externalising is therefore the most reliable defence against long-session decay.
• It is mutable. Context is append-only, so “updating” a plan means adding a second version while the first stays visible — the superseded-state problem in its purest form. A file has one current version, and the stale one is genuinely gone.
• It is addressable. The agent can read section three without paying for sections one, two and four. Nothing in a window supports partial access.
• It is inspectable and diffable. You can open the file mid-run, watch it change, diff it between steps, and check it into version control. Debugging an agent by reading its notes is considerably easier than reconstructing them from a 90,000-token trace.

The tool surface

The tools matter more than the store. A scratchpad exposed only as read-whole and write-whole re-imports every problem it was meant to solve, because every read costs the full document and every write requires the agent to reproduce it.

list()                        -> [{ name, bytes, modified }]
read(name, section?)          -> string          // partial, not whole
write(name, content)          -> { bytes }
append(name, content)         -> { bytes }       // cheap incremental note
replace(name, find, with)     -> { changed: n }  // edit without re-emitting
outline(name)                 -> [heading, ...]  // cheapest possible read

outline is the one most often missing and the one that changes behaviour most: it lets the agent decide whether to read at all for a handful of tokens. replace matters because the alternative to in-place editing is the agent regenerating a 20,000-token document to change one line, which is billed at the output rate and is the most expensive thing an agent can do by accident.

Two conventions make the store usable rather than a dumping ground. Give files predictable names the agent is told about up front — plan.md, findings.md, state.json — because an agent that invents filenames will fail to find them again three turns later. And keep a small always-resident index of what exists: the file list is cheap, and an agent that does not know a file exists will not read it.

When the window is still right

Externalisation is not free and the failure modes are real. The agent has to remember to read, which is a behaviour rather than a guarantee — the classic failure is an agent that writes a careful plan and then never opens it again, proceeding from a half-memory of what it wrote. Each read is a round trip. And a scratchpad is state that can be stale, contradictory or corrupted in ways a transcript cannot.

So: keep in the window anything small, anything needed on nearly every turn, and anything whose absence would change behaviour rather than quality. Put in files anything large, anything intermittent, anything that must survive compaction, and anything you will want to read yourself when the run goes wrong. For a coding agent this maps almost exactly onto the repo map and file window distinction, which is the pattern at its most developed.