Migrating Idempotency and Deduplication Logic Between Providers

Idempotency is the difference between a timeout costing you a retry and a timeout costing you two charged completions and two support emails to the same customer. It is also the capability most likely to be present on the provider you are leaving and absent on the one you are moving to.

What the key actually protects

An idempotency key is a client-generated identifier sent with a request. The server records the outcome against that key for some window, and a second request with the same key returns the recorded outcome instead of doing the work again. OpenAI’s official SDKs expose this as a per-request option that sets an Idempotency-Key header.

What it protects is narrow and worth stating precisely: it protects against your uncertainty about whether the first request arrived. The dangerous case is not an error response — an error you received is an answer. The dangerous case is a connection reset or a client-side timeout, where the request may have been fully processed and the response lost on the way back. Without a key, your retry is a second, independent generation: billed separately, potentially different in content, and if it triggered a downstream write, done twice.

Three properties decide whether a provider’s implementation is useful to you, and they are the three to look up in the target’s reference before assuming parity:

• The retention window. How long the outcome is remembered. If it is shorter than your maximum retry horizon — including retries from a queue that was paused overnight — the key stops protecting you exactly when you most need it.
• The conflict behaviour. What happens if you reuse a key with a different body. Returning the original result silently and rejecting with an error are both defensible, and they require different client code.
• The scope. Whether keys are namespaced per key, per project or per organisation, which decides whether a collision between two services is possible.

Why an unsupported key fails silently

This is the migration hazard, and it is a property of HTTP rather than of any particular vendor. Servers ignore headers they do not understand. If your client sends Idempotency-Key to a provider that has no such feature, nothing rejects it: the request succeeds, the header is discarded, and your retry logic goes on believing it is protected. There is no error to catch and no log line to notice.

So the audit step is not “does the code still send the header” but “does the target document honouring it”. Verify positively rather than by absence of failure: send the same key twice with a deliberately non-deterministic prompt and compare the two responses. If the second response carries a different response id and different content, the key is being ignored, whatever the documentation implied.

KEY="probe-$(uuidgen)"
for i in 1 2; do
  curl -s https://api.example.com/v1/chat/completions \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -H "Idempotency-Key: $KEY" \
    -d '{"model":"MODEL","temperature":1,
         "messages":[{"role":"user","content":"Say one random word."}]}' \
    | python -c 'import json,sys; d=json.load(sys.stdin); print(d.get("id"), d["choices"][0]["message"]["content"])'
done

Two identical ids and identical text means the key is honoured. Two different ids means you are now responsible for deduplication yourself, and you should also expect to be billed for both — a replayed response is generally not re-billed, whereas two real generations are two generations.

The client-side replacement

Where the target has no idempotency support, you rebuild it in front of the call. The shape is a small durable table and a state machine, and the important design decision is that the record is written before the request goes out.

create table llm_call_dedup (
  idem_key      text primary key,   -- derived, not random: see below
  state         text not null,      -- 'in_flight' | 'succeeded' | 'failed'
  request_hash  text not null,      -- sha256 of the canonicalised request body
  response      jsonb,              -- the completed result, replayed on retry
  attempt       int  not null default 1,
  created_at    timestamptz not null default now(),
  expires_at    timestamptz not null
);

1. Derive the key from the work, not from the attempt. A random UUID generated inside the retry loop changes on every attempt and protects nothing. Use a stable business identifier — the message id, the job id, the document version — hashed together with the request body.
2. Insert the row in state in_flight before sending. If the insert conflicts, another attempt owns this work: either wait for it or return its recorded response.
3. Send the request. On success, update the row to succeeded with the response body in the same transaction as any downstream write it causes.
4. On an ambiguous failure — timeout, connection reset — leave the row in_flight and increment attempt. The row is now the record that this work may already have happened, which is the information a bare retry throws away.
5. Expire rows on a window at least as long as your longest retry horizon plus the longest a job can sit in a queue.

The awkward part is honest to state: this protects your side effects, and it does not protect your bill. A generation that completed on the provider before the connection dropped is charged whether or not you received it. Client-side dedup makes the second attempt cheap only when the first attempt’s response was actually recorded, which it was not in the ambiguous case. That is the real cost of migrating to a provider without server-side idempotency, and it is worth quantifying from your own ambiguous-failure rate before the decision rather than after.

The part that really needs it

Plain completions are usually safe to repeat. The requests that are not safe to repeat are the ones whose results cause writes: a tool call that issues a refund, a generated email that gets sent, a batch job that appends rows. In practice the idempotency you need is at the tool boundary rather than at the model boundary.

So carry the key downward. When the model returns a tool call, derive the tool invocation’s idempotency key deterministically from the call — the provider’s tool-call id combined with your request’s key — and pass it to whatever the tool calls. A retried model request that produces the same tool call then produces the same downstream key, and the downstream system rejects the duplicate. This is the arrangement tested in retrying a tool call without a duplicate side effect, and it is worth confirming during the migration that the tool-call id field on the new provider is stable enough to key on — if it is not, derive the key from the tool name and canonicalised arguments instead.

Streams and partial responses

Idempotency and streaming interact badly and the interaction is often missed. A stream that drops halfway has produced real output and real billing, and there is no general mechanism for resuming it: reconnecting starts a new generation. So a retry of a dropped stream is not idempotent even where the provider supports keys for unary calls, because the first attempt’s partial output already exists on your side.

Two things follow. First, persist partial output as it arrives, keyed by the same dedup row, so a retry can decide whether to resume the user-visible task from what it already has rather than starting over. Second, make the completion of the stream — not its start — the event that commits any downstream effect, so a dropped stream leaves nothing half-done. The reconnection mechanics themselves are covered in testing SSE reconnect logic.