Debugging a Context Problem

“The model ignored my instruction” is four different bugs wearing the same symptom. There is an order in which to rule them out, and the first step answers most cases before any theorising starts.

The order matters

The instinct on a bad answer is to rewrite the prompt. It is the fastest thing to try, so it gets tried first, and it is the least likely cause in any system that has been running for more than a few weeks — the prompt was working yesterday. Rewriting it also destroys evidence: change the wording and you can no longer tell whether the original was ever in the window.

So the order below runs cheapest and most-likely first, and every step is an observation rather than a change. Do not modify anything until the step that fails tells you what to modify.

One preliminary saves time on about a third of these investigations: establish whether the same request fails in a fresh session. If it works when it is the first thing in an empty window, the material and the wording are both fine and the cause is somewhere in accumulation — which collapses the search space enormously before you have read a single log line.

Step 0: dump the context

Before any question can be answered you need to see, exactly, what was sent. Not the inputs to your assembler — the serialised payload. This is one small helper and it will pay for itself the first time it runs.

function dumpContext(messages, meta) {
  const rows = messages.map((m, i) => ({
    i,
    role: m.role,
    kind: m.meta?.kind ?? "?",        // system | tools | history | retrieved
    tokens: count(render(m)),
    head: render(m).slice(0, 60).replace(/\s+/g, " "),
  }));

  const total = rows.reduce((s, r) => s + r.tokens, 0);
  console.table(rows);
  console.log({
    total,
    window: meta.window,
    reserved: meta.reservedOutput,
    headroom: meta.window - meta.reservedOutput - total,
    prefixHash: sha1(render(messages).slice(0, meta.prefixChars)),
    byKind: groupSum(rows, "kind"),
  });

  fs.writeFileSync("ctx-" + meta.requestId + ".txt", render(messages));
}

Three outputs, each answering a different class of question. The table shows what is present and in what order. byKind shows where the window went, which is usually a surprise — the answer is very often “62% tool results.” And the full dump on disk is what you grep in step 1. Run it behind a flag on every request in development and on sampled requests in production.

The five questions

1. Is the material in the context at all?

Grep the dump for a distinctive phrase from the thing the model supposedly ignored. This eliminates or confirms the single most common cause in one command. If it is absent, stop — you have a retrieval, allocation or eviction bug, and no amount of prompt work will help. If it is present, the prompt was delivered and something else is wrong.

When it is absent, the dump usually tells you why immediately: the block is at zero tokens (the allocator dropped it below its floor), the block is truncated mid-way (it was shrunk rather than dropped), or the block is missing entirely (eligibility filtered it out or the gather step never produced it).

2. Is it present but contradicted?

Search for the other values too. If the deadline appears three times with two different dates, the model is not ignoring anything — it is choosing, and you gave it no basis for choosing. This is the dominant failure in long sessions and it is invisible unless you look for it specifically, because each individual statement looks correct. The fix is supersession rather than accumulation.

3. Is it present, uncontradicted, and badly positioned?

Check where it landed. If it is 60% of the way through a 200,000-token input, it is in the region the position literature identifies as least reliably used. This is testable in one move: re-run the identical request with the material moved to the end. If the answer changes, you have an ordering problem, not a wording problem. If it does not, continue down the list.

4. Is it present, well-positioned, and drowned?

Look at the ratio. A 200-token instruction against 180,000 tokens of material is competing badly, and a system prompt whose share has fallen by a factor of forty since turn one is the dilution case. Test it by re-running the same request with the material cut to a fifth. If behaviour returns, the instruction was fine and the context was too full.

5. Everything checks out and it still fails

Now, and only now, is it a prompt or model problem. Reproduce it in isolation — the exact material, the exact instruction, nothing else — and if it still fails at 2,000 tokens with nothing competing, you have a clean, minimal reproduction that belongs to the prompt engineering side of the house, or is a genuine capability limit. Reaching step 5 with a minimal repro is itself a good outcome; most investigations that start here never build one.

Failure signatures

Some symptoms map to causes reliably enough to shortcut the order.

Making it debuggable next time

• Persist the rendered context, addressable by request id. The single highest-value piece of instrumentation here. Without it, every investigation begins with a reconstruction that may not match what was sent.
• Log the per-block token breakdown on every request. It is a handful of numbers and it turns “the context is too big” into “tool results are 62% and grew 4× on Tuesday.”
• Log every drop and truncation as a warning. If your allocator drops a block below its floor, that is an event worth seeing. Silent degradation is the whole problem.
• Give every block a kind and every retrieved item an id. Anonymous text cannot be attributed, counted or searched for. Identifiers are what make a dump answerable.
• Assert the maximum context in a test. Render the largest realistic assembly and assert it fits with margin. Catches the regression at the commit that caused it rather than in an incident.