Testing That an Agent's Memory Persists Correctly Across Turns

An agent with memory has two moving parts: a model, which is non-deterministic, and a store, which is not. Every useful assertion about memory is about the store. Once you accept that, the test becomes an ordinary state-machine test and stops needing a model at all.

Assert on the store, not the answer

The tempting test is to say something in turn one, ask about it in turn five, and assert the reply contains it. That test fails for two unrelated reasons — the memory was lost, or the model chose to answer differently — and it cannot tell you which. It also costs a live request per turn, so it is slow enough that people run it rarely enough that it rots.

What is actually in question is narrower and entirely deterministic: after turn n, does the store contain what it should, in the right order, under the right key, and does the next request built from it include what it should? Both sides are values you can read. The model in the middle is irrelevant to the property, so replace it.

This is worth stating as a rule because it generalises past memory. Whenever a test involves a model, ask which half of the behaviour is yours. The store is yours. The trimming policy is yours. The order of messages, the session key, the tool result plumbing, the retry — all yours, all deterministic, all testable at unit speed. What the model says is not, and a suite that keeps trying to assert on it ends up slow, flaky and eventually disabled. Almost every page in this cluster is an instance of the same move.

A recording stub instead of a model

The stub does two jobs: it returns scripted replies so the conversation advances deterministically, and it records the exact request it was handed so you can assert on what the agent believed at that moment. Keep both in one object.

// stub-model.ts
export type Turn = { messages: unknown[]; toolCalls?: unknown[] };

export function stubModel(replies: string[]) {
  const seen: Turn[] = [];
  let i = 0;
  return {
    seen,
    async complete({ messages }: { messages: unknown[] }) {
      seen.push({ messages: structuredClone(messages) });
      const content = replies[i++] ?? "ok";
      return { role: "assistant" as const, content };
    },
  };
}

The structuredClone is not decoration. Most agent loops mutate the same array in place between turns, so a stub that stores the reference records the final state three times and every assertion about turn one passes for the wrong reason. This is the single most common defect in a hand-rolled recording stub, and it makes the test suite look green while the memory is broken.

Four invariants a memory store must hold

• Append-and-read-back is ordered. After three user turns, loading the session returns six messages alternating user and assistant, in the order they happened. Assert the roles as a sequence, not just the length — a store that appends the assistant reply before the user message passes a length check and produces nonsense on the next turn.
• The write happens before the reply is returned. If the agent answers the user and then persists, a crash between the two loses a turn the user has already seen. Assert the ordering by making the store throw and checking the user never received an answer.
• Sessions are isolated. Write under session A, load session B, get nothing. This is the invariant with the worst failure mode — one user reading another user’s conversation — and it is a one-line test, so there is no excuse for not having it.
• Replay is idempotent. A retried request must not append the same user message twice. Give each inbound message a client-supplied id and assert that appending it twice leaves the store with one copy.

import { beforeEach, describe, expect, it } from "vitest";
import { Agent } from "../src/agent";
import { MemoryStore } from "../src/memory";
import { stubModel } from "./stub-model";

describe("agent memory", () => {
  let store: MemoryStore;
  beforeEach(() => { store = new MemoryStore(); });

  it("records both sides of every turn in order", async () => {
    const model = stubModel(["one", "two", "three"]);
    const agent = new Agent({ model, store });

    for (const text of ["first", "second", "third"]) {
      await agent.send("session-a", text);
    }

    const saved = await store.load("session-a");
    expect(saved.map((m) => m.role)).toEqual([
      "user", "assistant", "user", "assistant", "user", "assistant",
    ]);
    expect(saved[0].content).toBe("first");
    // turn three was built from everything before it
    expect(model.seen[2].messages).toHaveLength(5);
  });

  it("does not leak between sessions", async () => {
    const agent = new Agent({ model: stubModel(["ok"]), store });
    await agent.send("session-a", "my card ends 4242");
    expect(await store.load("session-b")).toEqual([]);
  });

  it("appends a replayed message once", async () => {
    const agent = new Agent({ model: stubModel(["ok", "ok"]), store });
    await agent.send("session-a", "hello", { messageId: "m-1" });
    await agent.send("session-a", "hello", { messageId: "m-1" });
    const saved = await store.load("session-a");
    expect(saved.filter((m) => m.role === "user")).toHaveLength(1);
  });
});

Proving it survives a restart

“Persists” is the word in the requirement and it is the one an in-process test quietly skips. An agent holding a Map keyed by session id passes every test above and loses everything on deploy. The test that catches it constructs a second store instance over the same backing storage and reads through that, with no shared memory between them.

it("survives a process restart", async () => {
  const first = new MemoryStore({ url: process.env.TEST_REDIS_URL });
  await new Agent({ model: stubModel(["ok"]), store: first })
    .send("session-a", "remember: the invoice is 88031");
  await first.close();

  const second = new MemoryStore({ url: process.env.TEST_REDIS_URL });
  const saved = await second.load("session-a");
  expect(saved.map((m) => m.content)).toContain(
    "remember: the invoice is 88031",
  );
});

This is the one test in the set that wants a real backing service rather than a fake, because the thing under test is serialisation and durability, and an in-memory double has neither. Run it against a container started for the suite rather than against a shared development instance, so the test can flush the keyspace it uses without ruining somebody’s afternoon.

Where this test goes wrong

The usual failure is that the test asserts the store is correct while the agent no longer reads all of it. Once a conversation exceeds a token budget something has to trim or summarise, and a store that faithfully holds forty messages while the request carries eight is both correct and useless. That is a different property with a different test — assert on the outbound payload, not on the store — and the two belong in separate files so a failure tells you which layer moved.

The second failure is a stub that is too generous. If your stub always returns plain text, the memory path for tool calls is never exercised, and tool results are exactly the messages most likely to be dropped: they are large, they have an unusual role, and half the framework code that trims history treats them as expendable. Script at least one turn in which the stub returns a tool call, and assert the tool result is in the store and in the next request with its tool_call_id intact. A tool result whose id has been lost is a request most providers reject outright.

The third is scope creep into quality. Once the harness exists it is tempting to add a case asserting the agent uses the remembered fact, and that case belongs in an eval rather than in a unit suite: it is judging the model, it needs many samples to mean anything, and it can fail on a model upgrade that improved the product. Keep this file to structural claims about the store and the payload, and let a golden dataset carry the question of whether the answers are any good. The two run at different frequencies for good reasons — one on every commit, the other when the model or the prompt changes.