Idempotency for Expensive Operations

Idempotency is usually taught with payments, because a duplicated charge is obviously bad. A duplicated generation is the same problem with a smaller number attached and a worse detection story: it costs money, produces a different answer, and leaves no reconciliation trail that would make anyone look.

Why this operation is different

The classical definition — an operation is idempotent if performing it twice has the same effect as performing it once — assumes the effect is a state change you control. A model call has two effects. The one you wanted is a result. The one you did not is a charge on an account at a company you do not run, recorded in a ledger you can read but cannot write.

That second effect cannot be made idempotent by you. If the request reached the provider and the provider generated tokens, you are billed whether or not the response reached you. So the goal is narrower and more achievable: make sure that your system issues the call at most once per logical operation, and that every subsequent attempt returns the first result rather than producing a second one.

The non-determinism raises the stakes. With a deterministic dependency a duplicate call is merely waste — you get the same answer twice. Here the two answers differ, so if a duplicate slips through you must also decide which one is real, and any part of your system that read the loser is now inconsistent with any part that read the winner.

The race, written out

The obvious implementation is check-then-act: look for a record with this key, and if there is none, do the work and save it. Here is what two concurrent requests do to that. Both carry the same key, because a client retried after a timeout while the first request was still in flight.

  time   request A                      request B
  ----   ---------------------------    ---------------------------
   t0    SELECT ... WHERE key = 'k'
   t1    -> no row
   t2                                   SELECT ... WHERE key = 'k'
   t3                                   -> no row          <-- still nothing
   t4    call model .................
   t5                                   call model ........
   t9    -> result A  (billed)
  t10                                   -> result B  (billed)
  t11    INSERT (key='k', result A)
  t12                                   INSERT (key='k', result B)  <-- or UPDATE
                                        two charges, two answers, one key

The window between t1 and t11 is the entire duration of the model call, which is seconds. This is not a narrow race that needs unlucky scheduling; it is a several-second window that any impatient client, any retrying proxy and any at-least-once queue will find. The reason idempotency bugs are rarer with fast dependencies is simply that the window is measured in milliseconds there.

Claim before call

The fix is to make the durable write happen before the expensive call, not after it, and to let the database arbitrate. Insert a row in a claimed state with a unique constraint on the key. Exactly one caller wins the insert; everyone else gets a constraint violation and knows, without asking, that someone is already doing the work.

async function once(key: string, fingerprint: string, work: () => Promise<Result>) {
  const claimed = await db.query(
    "insert into idem (key, fingerprint, state, claimed_at) " +
    "values ($1, $2, 'claimed', now()) " +
    "on conflict (key) do nothing returning key",
    [key, fingerprint],
  );

  if (claimed.rowCount === 0) {
    const row = await db.one("select * from idem where key = $1", [key]);
    if (row.fingerprint !== fingerprint) throw new KeyReused();       // 422
    if (row.state === "succeeded") return row.result;                 // replay
    if (row.state === "failed") throw new PriorFailure(row.error);
    throw new InProgress(row.claimed_at);                             // 409
  }

  try {
    const result = await work();
    await db.query(
      "update idem set state='succeeded', result=$2, done_at=now() where key=$1",
      [key, result],
    );
    return result;
  } catch (error) {
    await db.query(
      "update idem set state='failed', error=$2, done_at=now() where key=$1",
      [key, classify(error)],
    );
    throw error;
  }
}

Two details carry most of the weight. The on conflict do nothing returning makes the claim a single atomic statement, so there is no window at all — the unique index is the lock. And the fingerprint, a hash of the request body, catches the client that reused a key with different content, which is a bug on their side that you want to report as a 422 rather than silently answer with someone else’s result.

Schema and the four responses

The expired-claim case is the one that turns a correct design into a production outage if you leave it out. A worker that is killed between claim and completion leaves the key permanently locked, and every subsequent retry of that logical operation gets a 409 forever. A claim needs a lease, and the lease needs to be longer than the longest possible attempt.

Where the key comes from

A key must be stable across retries of the same logical operation and different across genuinely different ones. That means it is generated by the party that decides an operation exists — usually the client, once, before the first attempt, and reused on every retry of that attempt.

• A random UUID minted at the top of the operation is the default and it is correct. The client stores it with whatever it is retrying.
• A natural key — summary:doc_id:content_hash — is better where one exists, because it deduplicates across independent callers and survives a client that lost its state. It also gives you a free cache: a second request for the same document revision replays instead of regenerating.
• Never derive it from the prompt alone if the prompt omits something that changes the answer. Model id, model version, temperature and your prompt template version all belong in the fingerprint, or an upgrade silently replays yesterday’s output.
• Never derive it from a timestamp or a request id generated per attempt. That is a new key every retry, which is the same as having none.

Expiry, and what you keep

Keys do not need to live forever, but they must live longer than the longest retry window of anything that might resend — including a user’s browser, your own queue’s maximum backoff, and a human clicking a button again ten minutes later. Twenty-four hours is a common floor. Sweep with a scheduled delete on done_at, and keep the index on the key rather than on the timestamp, because the hot path is the lookup and the sweep can afford a scan.

Keep the cost alongside the result. A row that records what the operation cost, which model produced it and how many attempts it took turns your idempotency table into the only place in the system that can answer “what did we spend on this document” without a join across three services.