Testing a Multi-Turn Tool-Calling Conversation End to End

A single-turn test tells you the loop can call a tool. It does not tell you the loop can hold a conversation: that the second turn still has the first turn’s result, that ids line up, that the third tool call was made with information only the second tool could have supplied. Those are properties of a whole transcript, so the fixture has to be a whole transcript.

The script is the test

Write down what the model returns on each turn, in order, and hand that list to a fake. Everything else in the test is derived from it. The script is data, so it lives in its own file and can be read by somebody who has never seen the loop.

// tests/fixtures/refund-conversation.ts
export const script = [
  { kind: "tool_call", tool: "get_order",
    args: { order_id: "55219" }, call: "c1" },
  { kind: "tool_call", tool: "get_refund_policy",
    args: { sku: "MUG-01", purchased_at: "2026-07-30" }, call: "c2" },
  { kind: "tool_call", tool: "start_refund",
    args: { order_id: "55219", amount_cents: 1499, reason: "damaged" }, call: "c3" },
  { kind: "text", text: "I've started a refund of EUR 14.99 for order 55219." },
] as const;

Notice what the script does not contain: the tool results. Those come from stubs registered in the test, because they are the other half of the contract and you want to vary them independently. A script that bakes in both sides can only ever replay one story; a script of model turns plus a set of tool stubs gives you every combination of the two.

The call field is a short stable identifier rather than a realistic provider id. Realistic ids are long, random and different on every recording, which makes a fixture diff unreadable and a fixture rewrite look like a change. Nothing in either API cares what the string is, only that the result carries the same one.

A player that fails loudly

The fake that plays the script needs two behaviours the naive version lacks: it must record what it was called with, and it must fail rather than return undefined when the loop asks for more turns than the script has.

function player(script: readonly any[]) {
  const requests: any[][] = [];
  let turn = 0;

  const model = async (messages: any[]) => {
    requests.push(structuredClone(messages));
    const step = script[turn++];
    if (!step) {
      throw new Error(
        `model called ${turn} times but script has ${script.length} turns`,
      );
    }
    return step.kind === "text"
      ? { choices: [{ finish_reason: "stop",
          message: { role: "assistant", content: step.text } }] }
      : { choices: [{ finish_reason: "tool_calls",
          message: { role: "assistant", content: null, tool_calls: [{
            id: step.call, type: "function",
            function: { name: step.tool, arguments: JSON.stringify(step.args) },
          }] } }] };
  };

  return { model, requests, turnsUsed: () => turn };
}

The thrown error is the important line. A loop with an off-by-one in its stop condition asks for one turn too many; a player that returns undefined makes that surface as a confusing property access somewhere deep in the loop, while a player that throws names the problem in the message. Assert turnsUsed() at the end too, so a loop that stops one turn early is caught as well as one that runs long.

Assert on the transcript

The final sentence came from your script, so asserting on it proves nothing. What you did not write is the message array the loop built, and that is where every interesting bug shows up.

import { describe, it, expect, vi } from "vitest";
import { script } from "./fixtures/refund-conversation";

it("runs the refund conversation to completion", async () => {
  const tools = {
    get_order: vi.fn(async () => ({ sku: "MUG-01", total_cents: 1499,
      purchased_at: "2026-07-30", status: "delivered" })),
    get_refund_policy: vi.fn(async () => ({ eligible: true, window_days: 30 })),
    start_refund: vi.fn(async () => ({ refund_id: "rf_88", state: "pending" })),
  };
  const { model, requests, turnsUsed } = player(script);

  const res = await runAgent("order 55219 arrived smashed", { model, tools });

  expect(res.status).toBe("complete");
  expect(turnsUsed()).toBe(4);

  // Tools ran in the scripted order, once each.
  const order = [
    ...tools.get_order.mock.invocationCallOrder,
    ...tools.get_refund_policy.mock.invocationCallOrder,
    ...tools.start_refund.mock.invocationCallOrder,
  ];
  expect(order).toEqual([...order].sort((a, b) => a - b));

  // Every tool call was answered, exactly once, with its own id.
  const last = requests[requests.length - 1];
  const calls = last.filter((m: any) => m.role === "assistant" && m.tool_calls)
    .flatMap((m: any) => m.tool_calls.map((c: any) => c.id));
  const results = last.filter((m: any) => m.role === "tool")
    .map((m: any) => m.tool_call_id);
  expect(results).toEqual(calls);
});

The last assertion is the one that pays for the whole test. Comparing the ordered list of call ids against the ordered list of result ids catches a missing result, a duplicated result, a result attached to the wrong call, and results emitted before the assistant message that asked for them — four separate bugs, all of which produce a provider 400 in production and none of which a happy-path assertion notices.

Two more assertions earn the lines they take. That the user message appears exactly once and first — a loop that re-appends the original input each turn quietly grows the prompt every iteration. And that the argument to the dependent call matches the earlier tool’s output rather than merely being present: asserting sku: "MUG-01" against the value get_order actually returned is what proves the second turn read the first turn’s result, instead of guessing something plausible from the user’s wording.

The turns worth scripting

• A turn that depends on the previous result. In the script above, get_refund_policy is called with a sku that only get_order could have supplied. Assert that argument explicitly: it is the only assertion that proves information flowed forward rather than being invented.
• A tool that fails mid-conversation. Have get_refund_policy reject on the second turn and script a recovery turn after it. The loop must still produce a result message for that call, and the conversation must continue.
• A text turn in the middle. Models emit prose between tool calls. A loop that treats any text as completion stops early here, which is the failure the stop-condition test exists for.
• The same tool twice with different arguments. Two lookups in one conversation is where id threading breaks, because both results look alike and a loop matching on tool name rather than id will happily pair them the wrong way round.

Recording a script from a real run

Hand-writing scripts is fine for three of them and tedious for thirty. The alternative is to run the conversation once against a real model with logging on, and write the assistant turns out to a file in the same format — a small script that reads your request log and emits the fixture is usually twenty lines.

1. Run the conversation once with a wrapper that appends every response to a JSON array.
2. Replace the provider ids with short sequential ones (c1, c2) and strip timestamps, usage counts and request ids — everything that changes between identical runs.
3. Check that replaying the script reproduces the run you recorded before you assert anything about it. A script that does not replay cleanly is usually missing a turn, and finding that out now is much cheaper than debugging it as a test failure later.
4. Redact anything from the arguments that should not sit in a repository, then commit the file and read the diff. If it is not readable, the format is wrong, not the conversation.
5. Write the assertions by hand anyway. A recorded script plus generated assertions is a snapshot of current behaviour, which passes forever and tells you nothing.

Re-record deliberately and rarely. The format the recording lands in decides whether that is a five-minute review or an unreadable thousand-line diff.