Testing That Retrying a Failed Tool Call Doesn't Duplicate a Side Effect

The agent called send_refund, the call timed out, the agent tried again. Whether the customer got one refund or two depends on a decision you made months earlier, and this is the test that pins it.

A timeout is an unknown, not a failure

This is the fact the whole page rests on. When a request times out, the three possibilities are: the request never arrived, it arrived and failed, or it arrived and succeeded and the response was lost. You cannot distinguish them from the client, and no amount of logging on your side changes that — the information you need is on the other side of a connection that is gone.

So “do not retry side-effecting tools” is not a fix. It converts a possible duplicate into a certain failure: the refund that did go through leaves the agent believing it did not, and the agent tells the customer so. The only workable design is to retry and make the second attempt provably a no-op, which means the side effect has to be keyed by something both attempts agree on.

The reference implementation is Stripe’s. Its documentation describes saving the status code and body of the first request for a given idempotency key and returning that same saved result on later requests with the same key — including 500s — with keys removable after at least 24 hours, and an error if the same key arrives with different parameters. Those three behaviours are the contract to build against, and they are worth reading in Stripe’s idempotent requests reference before designing your own.

Where the key comes from

A random key generated at call time is worse than no key, because it is different on the retry and buys nothing while looking like a precaution. The key must be a deterministic function of the call’s identity, and in an agent loop there are two candidates:

• The tool call id. Providers assign an id to each tool call in an assistant message. If your retry re-sends the same tool call, the id is stable and it is the cleanest key.
• A hash of the canonicalised arguments. Needed when the retry goes back through the model, because a re-planned call gets a new id for the same intent, and keying on the id would let it through. Canonicalise first — sorted keys, normalised numbers, no insignificant whitespace — or two identical intents hash differently.

Use both, combined with a scope that bounds the window in which a repeat is a duplicate rather than a legitimate second action:

import hashlib, json

def idempotency_key(conversation_id: str, tool: str, args: dict) -> str:
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(
        f"{conversation_id}|{tool}|{canonical}".encode()
    ).hexdigest()
    return digest[:32]

Scoping to the conversation is the judgement call. Too wide — a key scoped to the customer — and a genuine second refund next week is swallowed. Too narrow — scoped to the agent step number — and the retry gets a different key. The conversation is usually the right unit because it is the boundary within which the same request twice is almost certainly one request.

One more property the key needs: it must be computed before the first attempt and carried, not recomputed on the retry. Recomputation looks equivalent and is not, because anything that varies between attempts — a timestamp inside the arguments, a re-serialisation with different key ordering, a value the agent regenerated — produces a different key from the same intent. Compute once when the tool call is first dispatched, store it on the attempt record, and pass the stored value on every retry. A test for this is one line: assert the key on attempt two equals the key on attempt one, which is the assertion in the section below.

The harness

You need a transport that fails in the specific way that creates the ambiguity: the effect lands, then the response is lost. A mock that simply raises before doing anything tests a different, easier case.

class FakeMailer:
    def __init__(self):
        self.sent = []
        self.keys = set()

    def send(self, key: str, to: str, body: str):
        # Record the effect first, then decide, so the test can
        # simulate "it worked but you never heard about it".
        if key in self.keys:
            return {"status": "duplicate", "id": key}
        self.keys.add(key)
        self.sent.append({"key": key, "to": to, "body": body})
        return {"status": "sent", "id": key}


class LosesFirstResponse:
    """Delegates, but throws away the first response."""
    def __init__(self, inner):
        self.inner = inner
        self.calls = 0

    def send(self, **kw):
        self.calls += 1
        result = self.inner.send(**kw)
        if self.calls == 1:
            raise TimeoutError("read timeout after 30s")
        return result

Both assertions, not one

The common mistake is asserting only that the transport was called once. That asserts you did not retry, which is the behaviour you are trying to move away from. Assert on the transport and the effect store separately, and expect different numbers from each:

def test_retry_does_not_duplicate_the_email(agent):
    mailer = FakeMailer()
    transport = LosesFirstResponse(mailer)
    agent.register_tool("send_receipt", transport.send)

    agent.run("send the receipt for order 4417 to the customer")

    assert transport.calls == 2      # we did retry
    assert len(mailer.sent) == 1     # the effect happened once
    assert len({m["key"] for m in mailer.sent}) == 1

1. Assert the retry happened. A test that passes because retries are disabled is testing your configuration, not your key.
2. Assert the effect store has one record. This is the assertion the page exists for.
3. Assert both attempts carried the same key. Without this, a test can pass because the second attempt happened to be deduplicated by something else, and will regress silently when that something else changes.
4. Assert what the agent was told. A deduplicated call must return a success-shaped result, not an error, or the agent retries a third time or apologises to the user for a refund it successfully sent.

Keys that are too coarse and too fine

A key that deduplicates everything passes the test above trivially, so the suite needs the opposite case as well. Two genuinely different calls in one conversation must both go through — two refunds for two different orders, two emails to two recipients:

def test_two_different_sends_both_happen(agent):
    mailer = FakeMailer()
    agent.register_tool("send_receipt", mailer.send)

    agent.run("send receipts for orders 4417 and 4418")

    assert len(mailer.sent) == 2

Then the too-fine direction: the model emitting the same tool call twice in one assistant message, which happens with parallel tool calling and produces a duplicate with no retry involved at all. Feed a recorded assistant message containing two identical calls and assert one effect. The same key handles both cases, which is the argument for deriving it from arguments rather than from a retry counter.

Finally, test expiry if your store has one. A key retained for 24 hours means a retry 25 hours later is a second effect, which is correct for a payment and wrong for a nightly job that runs every 26 hours. Assert the boundary with a clock you control rather than a sleep — and see testing that an async job retry does not duplicate its webhook for the same problem one layer out.