Testing a Multi-Turn Conversation for Context Loss Between Turns

The bug reads as “it forgets things”. That description is compatible with a model failing to use information it was given and with information never being sent, and those have nothing in common. The test that separates them looks at the request body, where the answer is unambiguous.

The symptom and the two possible causes

A user says their account number in turn one. By turn five the assistant asks for it again. Two stories fit. Either the number was in the request and the model did not attend to it — a prompt problem, fixed by structure or by restating salient facts near the end of the context — or the number was not in the request at all, which is a plain bug in your history handling and no amount of prompt engineering will touch it.

Teams routinely spend a week on the first story when they have the second. The reason is that the only artefact anybody looks at is the conversation transcript in the UI, which shows the whole history because the UI keeps its own copy. What left your process is a different object, and nobody has looked at it.

The payload is the evidence

So make the payload the thing under test. The property is stated without reference to model behaviour: for every turn after the one in which a fact was established, the serialised request sent to the provider contains that fact. It is checkable by string search, it is deterministic, and it fails loudly the moment a trimming policy starts eating the wrong end of the conversation.

A string search is cruder than it looks and that is a feature. You are not asserting the fact is well presented or in the right message — only that it has not vanished. If you want the stronger property, assert on the parsed body: that a message with the establishing id is still in messages, and that its content is byte-identical to what was stored. Start with the crude version, because it is the one that catches the bug people actually ship.

Capturing it with an interceptor

MSW is a good fit here because it intercepts at the network layer, so it sees what the vendor SDK actually sent rather than what you passed into it — which matters, since SDKs rename fields, drop undefined values and sometimes restructure messages. Its Node entry point is setupServer from msw/node, with handlers built from http and HttpResponse.

// capture.ts
import { http, HttpResponse } from "msw";
import { setupServer } from "msw/node";

export const captured: Array<Record<string, any>> = [];

export const server = setupServer(
  http.post("https://api.openai.com/v1/chat/completions", async ({ request }) => {
    const body = await request.json();
    captured.push(body as Record<string, any>);
    return HttpResponse.json({
      "id": "chatcmpl-test",
      "object": "chat.completion",
      created: 0,
      model: (body as any).model,
      choices: [{
        index: 0,
        message: { role: "assistant", content: "noted" },
        finish_reason: "stop",
      }],
      usage: { prompt_tokens: 10, completion_tokens: 2, total_tokens: 12 },
    });
  }),
);

// context-loss.test.ts
import { afterAll, afterEach, beforeAll, expect, it } from "vitest";
import { captured, server } from "./capture";
import { Agent } from "../src/agent";

beforeAll(() => server.listen({ onUnhandledRequest: "error" }));
afterEach(() => { server.resetHandlers(); captured.length = 0; });
afterAll(() => server.close());

it("still carries the account number at turn five", async () => {
  const agent = new Agent({ sessionId: "s-1" });
  await agent.send("My account is 55-01923.");
  for (const q of ["and my balance?", "since when?", "in euros?", "thanks"]) {
    await agent.send(q);
  }

  expect(captured).toHaveLength(5);
  const fifth = JSON.stringify(captured[4]);
  expect(fifth).toContain("55-01923");
});

Set onUnhandledRequest to "error". Without it, a request to a URL you did not write a handler for goes to the real internet, which turns a unit test into a billable one and, worse, makes it pass for reasons unrelated to your code.

What the trimmer actually dropped

When that test fails, the next question is which rule dropped the message. Three trimming policies are common and they fail differently. A fixed window of the last N messages drops turn one as soon as the conversation is long enough, deterministically, at exactly the same point every time — which is why this bug so often reproduces on the sixth turn and never on the fifth. A token-budget trimmer drops it at a point that depends on how verbose the intervening turns were, so it looks intermittent. A summarising compactor keeps a paraphrase and loses precisely the things a paraphrase loses: digits, identifiers, names, negations.

Parameterise the test over conversation length rather than picking one, and you find the boundary instead of a single anecdote. Run the same scripted conversation at 4, 8, 16 and 32 turns and assert the fact survives all of them; the first length that fails names the policy. It is also worth asserting the inverse property — that the payload does not grow without bound — because a history handler with no trimming at all passes every test in this section right up until it exceeds the context window in production. If you have a pre-flight size check, this is where its threshold gets exercised.

Pinning, and the test that proves it

The fix for all three policies is the same shape: mark some messages as unevictable, and give the trimmer a rule that removes only from the evictable set. In practice this means extracting durable facts into a small structured block that is rebuilt each turn and placed near the end of the prompt, rather than trusting a message from twenty turns ago to survive.

That design has its own test, and it is not the one above. Assert that the pinned block is present in every payload; assert that it is rebuilt from the store rather than copied forward, by mutating the store between turns and checking the payload follows; and assert that when the budget is exceeded it is the evictable messages that disappear. The general mechanics of losing the wrong end of a prompt are covered in prompt truncation; what belongs here is the assertion that your specific trimmer honours the pin.

One last property is worth an assertion and is almost always missing: that the pinned block is bounded. A facts block rebuilt from a store that keeps accumulating entries grows for the life of the session, and because it is unevictable by construction it eventually crowds out the conversation it was meant to support — the trimmer dutifully deletes every recent turn while preserving forty stale facts. Cap the block, assert the cap in the same test that asserts the pin, and decide explicitly what happens on overflow. Dropping the oldest fact is usually right; silently truncating the block mid-entry, which is what a naive character limit does, produces a half-written identifier that reads as authoritative and is wrong.