Context Windows and Big Repositories

Every year the window gets bigger and the answer stays the same: you are sending a small subset of the repository, so the entire problem is choosing it. What changes is only how small.

The scale problem, in tokens

At roughly ten tokens per line of source, a five-million-line monorepo is about fifty million tokens. A one-million-token window holds 2% of it. Doubling the window moves that to 4%, which changes nothing structural.

Cost and latency move in the other direction and faster. Filling a one-million-token window once, at an assumed $3 per million input tokens, is $3 per request before the model has generated a single token. Two hundred agent turns a day is $600 a day, and attention costs grow with the square of sequence length so the latency is not linear either. This is the arithmetic behind “just paste the repo” being both bad retrieval and expensive.

Quality does not rise monotonically with context either. Liu et al. (2023) found accuracy shaped like a U across position — strong at the start and end, weaker in the middle — so material buried in a large context is not reliably used at all. Effective context length is generally shorter than the advertised one.

Three layers, in this order

Most teams reach for embeddings first. For code that is the wrong end of the list, because code has two properties prose does not: its identifiers are exact strings, and it has a real dependency graph.

Layer 1 — lexical. Exact, and usually sufficient

“Where is resolveTaxRule defined and who calls it” is answered perfectly by ripgrep. A symbol name is a unique token; semantic similarity can only degrade an exact match. Lexical search is also instant, has no index to keep fresh, and needs no infrastructure — which means it is what an agent should reach for first, and what you should give it a tool for.

rg -n --type ts 'resolveTaxRule' --glob '!**/dist/**' -C 2
rg -l 'implements PaymentProvider'          # find the implementations
rg -n 'TODO|FIXME|XXX' src/billing/         # local hazards worth knowing

Layer 2 — structural. The graph you already have

A language server knows precisely what a lexical search can only guess: definitions, references, implementations, and the type of an expression. textDocument/definition, textDocument/references and callHierarchy/incomingCalls give exact answers where grep gives candidates — no false positives from a same-named method on a different class, no misses from a re-export.

Exposing an LSP client as an agent tool is more work than exposing grep and it is the highest-value tool you can add for a typed codebase, because “every caller of this function” is the exact question that decides whether a change is safe.

Layer 3 — semantic. For the questions with no keyword

Embeddings earn their place on one query shape: “where is authentication handled”, asked by someone who does not know that the module is called gatekeeper. That is a genuine gap in the first two layers and worth an index for.

Embed summaries and signatures rather than raw bodies — a natural language description of what a function does embeds far closer to a natural language question than its implementation does. And keep the index scoped: it is a discovery aid for humans and agents, not the primary retrieval path. Retrieval over code goes into the indexing choices in detail, and index freshness is the operational problem it creates.

Chunking code is not chunking prose

Fixed-size chunking with overlap is standard for documents and actively harmful for source. A 512-token window cuts a function in half, and half a function retrieved without its signature, its imports or its enclosing class is close to useless — the model cannot tell whatthis is, where the helper comes from, or what the types are.

Chunk on syntax instead. Parse with tree-sitter, take function and class declarations as chunk boundaries, and prepend the context that makes a fragment interpretable:

# chunk header, ~30 tokens, makes the body interpretable on its own
# file:  src/billing/invoice.ts
# scope: class InvoiceService
# imports: TaxRule from ./tax, Cents from ../money, db from ../db
# ---
  async issueInvoice(orgId: OrgId, period: Period): Promise<Invoice> {
    ...
  }

Split a function that exceeds the chunk size at statement boundaries and repeat the header, never mid-expression. And index the file’s path as searchable text: paths carry real meaning in a well-organised repository — packages/billing/src/tax/vat.ts is a better relevance signal than most of its contents.

What a monorepo already knows

In a build-system monorepo the dependency graph is not something to infer — it is declared, complete and queryable, and it answers the question retrieval is usually approximating. What is the blast radius of a change to this target?

# everything that transitively depends on the billing library
bazel query 'rdeps(//..., //packages/billing:lib)' --output package

# the tests that a change to it could possibly break -> your agent's test cmd
bazel query 'kind(".*_test", rdeps(//..., //packages/billing:lib))'

# pnpm / nx / turbo equivalents
pnpm why @acme/billing      ·      nx graph --focus=billing

Two uses. First, that reverse-dependency set is the correct file set to retrieve for a change — better than any similarity score, because it is the ground truth of what could break. Second, the test query gives an agent a scoped test command instead of a full-suite run, which is usually the difference between a two-minute loop and a forty-minute one.

CODEOWNERS is the other underused signal: it partitions the repository into areas that were drawn by humans according to what belongs together. Restricting retrieval to the owning team’s subtree is a crude filter that removes an enormous amount of irrelevance for free.

Letting the model do the retrieving

One-shot retrieval has to guess what will be needed before anything has been read. An agent with grep, read_file with a line range, ls and ideally an LSP tool gets to iterate: find the symbol, read its definition, find its callers, read the one that looks wrong. That converges on the right context far more reliably, because each step is informed by the last.

The costs are real and worth stating. It takes turns, and turns are tokens — every round trip re-sends the transcript, which is why an exploration-heavy session is expensive out of proportion to its output. It can wander, so a turn budget is not optional. And it can be catastrophically slow if read_file has no line range and it opens a 4,000-line module to see one function.

The hybrid that works: pre-load the ranked signature index and the reverse-dependency file set, then let the model fetch bodies on demand. It starts with a map rather than a blank page, so it spends its turns reading the right things instead of finding out what exists.