Giving a Coding Model the Right Context

Context assembly is a budgeting problem before it is a retrieval problem, and the budget is far tighter than the headline window suggests. Do the arithmetic once and most of the design decisions make themselves.

The arithmetic nobody does

Code tokenizes at roughly three characters per token — worse than prose, because identifiers split and punctuation is dense. A source line averages around thirty useful characters, so call it ten tokens per line. That estimate is rough and it is rough in a direction that does not matter, because the conclusion survives a factor of two.

A one-million-line repository is therefore about ten million tokens. A 200,000-token window holds 2% of it. A one-million-token window holds 10%. There is no window on any roadmap that changes the shape of this problem: you are always choosing a small subset, and the only question is whether the choosing is done deliberately or by whatever files happened to be open.

Count it for your own repo before designing anything:

# lines of tracked source, excluding generated and vendored trees
git ls-files '*.ts' '*.tsx' '*.py' '*.go' \
  | grep -vE '(^|/)(dist|vendor|generated|__snapshots__)/' \
  | xargs wc -l | tail -1

# 412,904 lines  ->  ~4.1M tokens  ->  a 200k window is ~4.8% of it

A signature index, then a ranking

The move that buys the most is to send shapes rather than bodies: for each file, its path and the signature of every exported symbol, one line each. A signature line costs roughly fifteen to thirty tokens against several hundred for the function it names, so the same budget covers one or two orders of magnitude more of the codebase.

src/billing/invoice.ts
  export function priceLineItems(items: LineItem[], tax: TaxRule): Cents
  export function issueInvoice(orgId: OrgId, period: Period): Promise<Invoice>
  export class InvoiceLockedError extends DomainError

src/billing/tax.ts
  export function resolveTaxRule(country: string, vatId?: string): TaxRule

Extract it with tree-sitter or ctags — both give you the node kinds and the byte ranges, and neither needs the code to compile.

That index does not fit either. A repository with five thousand exported symbols costs around a hundred thousand tokens as a flat list, which is half a 200k window spent on a table of contents. So it has to be ranked and truncated, and the ranking that works is graph-based: build the import graph, seed it with the files the task actually mentions, and run a PageRank-style propagation so that files depended on by the seeds — and files that depend on them — score highly. This is the approach the aider project documents for its repo map, and it is worth copying because it captures the thing lexical search cannot: a file that never mentions your keyword but that every seed file imports is almost certainly relevant.

Then spend the remaining budget top-down: full bodies for the top few files, signatures for the next few dozen, nothing for the rest.

What earns its tokens

• The failing test and its exact output. Not “the test fails” — the assertion, the expected and actual values, and the stack. This is the highest-value block in the entire prompt because it is the only part that is unambiguously ground truth.
• The two or three files that will change, in full.
• The interface between them — the type, the schema, the protobuf. Interfaces are where a model’s guess is most expensive and cheapest to prevent.
• One nearby example of the pattern to follow. A single existing handler teaches your conventions better than a paragraph describing them, because it carries the imports, the error style and the naming all at once.
• Version pins for anything unusual. One line from the lockfile prevents a whole class of code written against a different major version.

What quietly eats the window

The largest files in most repositories are ones no human reads, and they get swept in by any “attach the folder” gesture. A package-lock.json in a mid-sized project runs comfortably past a hundred thousand tokens on its own. Generated API clients, snapshot test files, minified bundles, migration histories, vendored dependencies and .min.css are all in the same category: enormous, uninformative, and indistinguishable from real source to a naive file walker.

Keep an explicit ignore list next to your tooling rather than relying on .gitignore, because the problematic files are frequently tracked on purpose. The rule of thumb that catches most of it: if no human has edited the file by hand, it does not belong in a prompt.

The other quiet waste is sending the same thing twice. A file included because you attached it, again because the repo map picked it, and a third time because the agent read it two turns ago is three copies in one request — and worse than the token cost, the model now has three versions of a file it has since edited and no way to know which is current. Deduplicate by path before assembly, keep exactly one copy of any file, and make it the latest one. This is the single most common bug in a home-grown context assembler and it presents as the model “forgetting” an edit it made.

The subtler waste is history. Long chat transcripts carry stale premises — a file you already changed, an approach you abandoned — and the model keeps honouring them, which shows up as a session that gets worse as it gets longer.

Order and cacheability

Position matters. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023), found retrieval accuracy shaped like a U across position: material at the start and end of a long context is used far more reliably than material in the middle. So the instruction and the failing test go last, immediately before the model’s turn, and the bulk reference material goes in the middle where its exact position matters least. The effect and its limits are worth reading before you rely on a long window.

That ordering happens to coincide with the one caching wants. Prompt caches key on an exact prefix, so the layout is: stable material first (repo instruction file, signature index, conventions), then semi-stable (the files for this task), then volatile (the transcript and the instruction). Sort it the other way and every request is a cache miss on everything — the break-even arithmetic is unforgiving in an agent loop that re-sends its prefix dozens of times.