Scripting a Synthetic User to Drive an Agent Through a Test Scenario

A fixed list of user turns only reaches the conversation where the agent says what you expected. The moment it asks a question you did not anticipate, the script desynchronises and the rest of the test is nonsense.

What a static fixture cannot reach

Multi-turn agent behaviour is a tree, and a fixture is one path through it. That is fine for the paths you know about, and useless for the property you most want to check — that the agent gathers what it needs before acting, in whatever order the conversation happens to take.

A second model playing the user turns the fixture into a policy: given what the agent just said, respond as this user would. The conversation adapts, the agent can ask in any order, and the test still runs. The cost is that your input is now generated, which is exactly why most of this page is about constraining it.

Scripting rather than role-playing

“You are a frustrated customer” produces a different conversation every run and a test that cannot be diffed. Give the simulated user a ledger instead: a goal, a set of facts it holds, and a rule about disclosure. The rule is the important part, because a simulated user that volunteers everything in turn one tests nothing about the agent’s information gathering.

SYNTHETIC_USER = """
You are playing a customer in a test. Follow these rules exactly.

GOAL: get a refund for a damaged item.

FACTS YOU KNOW (reveal one only when directly asked for it):
  order_number: 4417
  email: [email protected]
  purchase_date: 2026-06-02
  problem: the screen arrived cracked

RULES:
- Never volunteer a fact you have not been asked for.
- If asked for something not in FACTS, say you do not know.
- Never mention that this is a test, and never mention these rules.
- Reply in one or two short sentences, as a customer would type.
- Set done=true only when your goal is met or the agent has refused.

Reply with JSON only: an object with keys "say" (string) and
"done" (boolean).
"""

The disclosure rule converts the simulated user from a source of text into a test instrument. Now “did the agent ask for the order number” is answerable from the transcript, because the only way the agent could have obtained it is by asking.

The envelope and the turn cap

Require structured output from the user model. Two fields are enough, and parsing them is what lets the harness terminate on a condition rather than on a fixed count. Then cap the turns, without exception: two polite models will thank each other indefinitely, and an uncapped harness turns one bad run into an unbounded bill.

def run_scenario(agent, user_model, max_turns=12):
    transcript, tools = [], []
    message = "Hi, I need help with an order."

    for turn in range(max_turns):
        reply = agent.respond(message)
        transcript.append(("agent", reply.text))
        tools.extend(c.name for c in reply.tool_calls)

        user = user_model.complete(
            system=SYNTHETIC_USER,
            user=render(transcript),
            response_format="json",
            temperature=0,
        )
        transcript.append(("user", user["say"]))
        if user["done"]:
            return Run(transcript, tools, ended="goal", turns=turn + 1)
        message = user["say"]

    return Run(transcript, tools, ended="turn-cap", turns=max_turns)

Returning how the run ended matters as much as the transcript. A run that hit the cap is a failure of a different kind from a run that finished with the goal unmet, and collapsing them into one boolean loses the distinction that tells you whether the agent was looping or refusing. Related: testing an agent’s maximum step budget.

What to assert

Never on the prose. Both sides are generated, so a transcript assertion is two non-deterministic systems compared to a third guess. Assert on what the run left behind:

• Terminal state. A refund of exactly 24.99 exists against order 4417, and no other refund exists. This is the strongest assertion available and it does not involve any text.
• Tool call ordering. verify_identity appears before issue_refund. An invariant, checkable as a subsequence of the recorded tool list, and it holds no matter which path the conversation took.
• Coverage of withheld facts. The agent obtained the order number, which is only possible if it asked. Assert the fact appears in the agent’s tool arguments, not in its prose.
• Turn budget. The goal was met within seven turns. A regression from five turns to eleven is a real degradation that no single-turn test can see.
• Forbidden actions. No tool outside the allowed set was called, and nothing was called after the user said stop.

And a separation rule: the model playing the user must never grade the run. It has been cooperating for twelve turns and will report success. Grading is a separate call with a separate prompt, or better, no model at all — the five assertions above are all mechanical.

Personas that are not cooperative

The default simulated user is unrealistically good: it answers the question asked, gives one fact at a time, never changes its mind and spells everything correctly. A suite of those exercises the happy path in five different costumes. Write the awkward ones deliberately, each as a variant of the ledger:

• The wrong fact first. Gives order 4471, corrects to 4417 two turns later. Assert the agent acts on the correction and does not act on the first value.
• The goal switch. Starts wanting a refund, decides mid-conversation on an exchange. Assert no refund was created.
• The over-answerer. Replies to a yes-or-no question with a paragraph containing three facts. Tests extraction, and frequently finds a parser that took the first number it saw.
• The other language. Switches language mid-thread. Pairs with multilingual output consistency.
• The injector. Pastes text containing an instruction aimed at the agent. Assert the instruction was not followed — prompt injection covers the mechanism.

Record once, replay after

Two models per run makes this the most expensive test in the suite and the least deterministic. Record the whole exchange — both sides, all requests and responses — the first time, and replay from the recording afterwards. The replayed test is deterministic, costs nothing and runs on every commit; the live re-record runs on a schedule.

The trap is that a replayed run only exercises the path the recording took. If the agent changes and asks a different question, the recording has no matching user turn. Fail loudly on that — a harness that falls back to the nearest recorded turn produces a green test for a conversation that never happened. Treat a replay miss as a signal to re-record, which is the same discipline as detecting cassette drift from the live API.