What Changes in Client Retry Logic When the API Holds State

Against a stateless endpoint, a retry after a timeout costs money and nothing else: the request is a pure function of data you hold, so sending it twice computes the same thing twice. Against an endpoint that stores the turn, the same retry can leave the conversation with the user’s message in it twice — and you will not see it until a later turn reads oddly.

Why stateless retries are cheap

When you assemble the whole context on every call, the request carries no dependency on anything the server remembers. If it times out, you do not know whether the model generated an answer, but you do know that the conversation on the server is exactly what it was before, because there is no conversation on the server. Re-sending produces a second independent generation of the same turn, and whichever reply you actually receive is the one you append. Your transcript can only be appended to by you.

That is the property doing all the work, and it is worth naming precisely: the request is idempotent with respect to state while being non-idempotent with respect to cost. So the stateless retry question is purely economic — how many duplicate generations are you willing to pay for, which is what the cost of retries and backoff testing cover.

Stateful endpoints break the first half of that sentence. If a call both generates and appends, then a call that succeeded server-side but failed to reach you has changed the world. Retrying is no longer recomputing; it is doing it again.

A failure taxonomy that is about commitment

The useful question is not “what status code” but “could the server have committed?” Grouping failures that way gives four classes, and only one of them is genuinely ambiguous.

• Never dispatched. DNS failure, connection refused, TLS failure, connect timeout, or a local error before the socket opened. The server never saw it. Safe to retry unconditionally, on any API.
• Dispatched and definitively rejected. A 400 with a validation message, a 401, a 404. The server answered and did not do the work. Safe, though retrying is usually pointless.
• Rate limited. A 429 before generation began. Normally safe, and the retry-after handling is the same as ever — but note that a 429 arriving after a partial stream is a different animal and belongs in the next class.
• Ambiguous. Read timeout after the request body was sent; a connection reset mid-response; a 502 or 504 from infrastructure in front of the model; a stream that stopped delivering without a terminal event. In every one of these the server may have completed and stored the turn. On a stateless API this class costs money; on a stateful one it is where duplicates come from.

Client libraries retry some of these for you, and their defaults are tuned for the stateless assumption. If you adopt server-side state, go and read what your client retries automatically before you write any logic of your own — a library-level automatic retry on a 500 is a duplicate turn you did not author, and it is precisely the kind of default change reading the changelog is meant to catch.

What actually duplicates

Three distinct things can be duplicated and they have different consequences, which is why “did it duplicate” is not one question.

The stored turn. The user message and the assistant reply both land in the conversation twice. The immediate symptom is usually nothing at all; the delayed symptom is that the next turn sees a transcript in which the user asked the same thing twice and the assistant answered twice, which reliably degrades the answer and looks like a model problem.

The side effect. If the turn included a tool call that your code executed, a duplicate turn can mean a second execution — a second refund issued, a second email sent. This is strictly worse than a duplicate message and it is why tool loops need their own guard regardless of the API shape; testing that a retried tool call has no duplicate side effect is the page for that half.

The charge. Both generations are billed. This is the only one of the three that also happens on a stateless API, and it is the least serious.

A fourth possibility is worth ruling in rather than out: a branch. If your retry continues from the same prior identifier, you have not appended twice to one thread — you have created two divergent continuations, and whichever identifier you happen to store becomes the real one while the other is orphaned but still billed and still stored. That is quieter than a duplicate and harder to spot.

The idempotency ledger

The pattern that fixes all of this is the one payments systems use, and the important property is that it works whether or not the provider offers anything. You generate a key for each logical turn, record your intent before dispatching, and make the record the thing that decides whether a retry is a new turn or a repeat of an existing one.

# one key per logical turn, generated before the first attempt
turn_key = uuid4().hex

# 1. write intent first, so a crash between here and the call is visible
db.insert_turn(conv_id, turn_key, status="pending",
               prev_response_id=conv.last_response_id)

try:
    resp = client.responses.create(
        model=MODEL,
        input=[{"role": "user", "content": user_text}],
        previous_response_id=conv.last_response_id,
        extra_headers={"Idempotency-Key": turn_key},   # honoured or ignored
    )
    db.complete_turn(turn_key, response_id=resp.id, status="done")
except AmbiguousFailure:
    db.mark_turn(turn_key, status="unknown")
    # do NOT retry here; see the next section

Three rules make this work, and each is load-bearing.

• The key is per logical turn, not per HTTP attempt.Generating a new key on retry defeats the entire mechanism. The key belongs to the user’s intent to say something, and it is reused by every attempt to say it.
• Intent is written before dispatch. A record written after the call cannot describe a call that never returned, which is exactly the case you are defending against.
• An ambiguous failure moves to a third state, not back. A turn is pending, done, or unknown. Collapsing unknown into failure is what produces duplicates; collapsing it into success is what produces silent data loss.

If the provider does honour an idempotency header, you get server-side deduplication for free and the ledger becomes a belt to the braces. Check the provider’s own reference for whether the header is supported and for how long keys are retained — a deduplication window measured in hours does not help a retry issued the next morning. Because that support varies and can change, the pattern above is written so the client-side ledger is sufficient on its own.

Reconcile, then retry

The final piece is what to do with an unknown turn. The answer is never to retry it blind. Ask the server what it has, then decide.

1. On the next request for that conversation, notice the unknown row before doing anything else.
2. Read the conversation state from the provider — retrieve the response by identifier if you captured one, or list the items that follow the last identifier you know landed.
3. Compare against your ledger. If a turn matching your key or your user text is present, mark the row done, adopt the response identifier the server reports, and continue. The first attempt succeeded and only the reply was lost.
4. If nothing is present, the request never committed. Retry with the same key and the same prior identifier.
5. If two matching turns are present, you already duplicated. Adopt one identifier as canonical and record the other as orphaned so the cost is attributable rather than mysterious.
6. Cap the whole thing. After a small number of ambiguous attempts, stop and surface the failure to the user rather than continuing to append to a conversation you no longer have an accurate picture of.

Streaming deserves one closing note, because it is where this bites most often. A stream that dies after the server committed is an ambiguous failure even though you received most of the answer, and reconnecting is not resumption unless the API explicitly supports resuming a specific response from a specific position. Where it does not, treat a dropped stream as an unknown turn and run the reconciliation above. Restarting the request instead is the single most common way a duplicate turn gets into a conversation, and it happens on exactly the network conditions that are hardest to reproduce.