Idempotency Keys for a Queued Model Request That Might Retry

Every queue in production use here is at-least-once. That is not a bug to be configured away, it is the delivery guarantee you are paying for. The job is to make the second delivery free, and the part that is usually done wrong is not the check — it is when the key is claimed.

Why the redelivery happens at all

There are four independent causes and a fix that only addresses one of them is not a fix. A visibility timeout or lock expires while the worker is still running. A worker crashes after calling the model but before acknowledging. The broker’s own delivery is duplicated, which SQS documents as possible even inside the visibility window. Or your producer retries a send whose response it never saw, putting two messages on the queue for one request.

The first two are worker-side, the third is broker-side, and the fourth is producer-side, which is why deduplication has to key on something the producer decided rather than on anything the broker generated. A message id deduplicates causes one to three and does nothing about cause four.

What makes this worth solving properly, rather than accepting as noise, is that the duplicated operation costs money and may have side effects. A duplicated cache warm is invisible. A duplicated generation is a line on an invoice and possibly a second email to a customer.

Choosing the key

The key must be generated by whatever first decided the work should happen, travel in the message body, and be stable across every retry of that decision. In practice that is one of three things: an identifier the caller supplies, a natural key from your own domain such as “summarise document 4471 at version 9”, or a hash of the request payload.

A hash is the tempting option and the one with a trap. If the payload includes a timestamp, a request id or a floating-point temperature that serialises inconsistently, the hash differs between the original and the retry and the whole mechanism silently does nothing. If you hash, hash a canonical subset of the fields that define the work, and write down which fields those are.

Do not use the broker’s message id. It is stable across redeliveries of one message and different for the producer’s duplicate send, which is precisely backwards from what you need.

The conditional write is the whole mechanism

The mechanism is one atomic operation: insert a row for the key if and only if no row exists. Every store worth using has this. In DynamoDB it is PutItem with a ConditionExpression of attribute_not_exists on the partition key, which fails with ConditionalCheckFailedException when the key is taken. In PostgreSQL it is an insert against a unique index, or INSERT ... ON CONFLICT DO NOTHING and a check of the affected row count. In Redis it is SET key value NX.

What must not happen is a read followed by a write. Two workers holding the same redelivered message will both read “absent” and both proceed; the window is small and the failure is exactly the one you were trying to prevent. The condition has to be evaluated by the store, in one operation.

import time, boto3
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")

def claim(key: str, lease_seconds: int) -> bool:
    now = int(time.time())
    try:
        ddb.put_item(
            TableName="idempotency",
            Item={
                "id":         {"S": key},
                "status":     {"S": "IN_PROGRESS"},
                "expiration": {"N": str(now + 86400)},
                "lease_until":{"N": str(now + lease_seconds)},
            },
            ConditionExpression=(
                "attribute_not_exists(id) OR "
                "(#s = :inprog AND lease_until < :now)"
            ),
            ExpressionAttributeNames={"#s": "status"},
            ExpressionAttributeValues={
                ":inprog": {"S": "IN_PROGRESS"},
                ":now":    {"N": str(now)},
            },
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

The in-progress window

The naive version writes the key after the model call succeeds. That leaves the entire duration of the call unprotected, which on this workload is the ninety seconds during which redelivery is most likely, because the reason for the redelivery is usually that the call took too long. Writing the key afterwards protects against everything except the case that actually happens.

So claim first, with a status of in-progress, then call the model, then update the row to done with the result attached. Three states, and each gets a different response on a duplicate delivery: done means return the stored result and acknowledge; absent means claim and proceed; in-progress means another worker holds it, so back off and let the message become visible again rather than acknowledging it away.

That third branch is why the claim above carries a lease. A worker that dies while in-progress would otherwise leave the key permanently claimed and the work permanently undone — the deadlock version of a duplicate. The condition allows a claim to be taken over once the lease has passed, which converts a crashed worker into a delay rather than a lost job.

If you are on Lambda, Powertools for AWS implements exactly this and is worth using instead of the above. Its @idempotent decorator wraps a handler, @idempotent_function wraps any function with a data_keyword_argument, and a DynamoDBPersistenceLayer stores records with documented default attribute names of id, expiration, in_progress_expiration, status and data. It raises IdempotencyAlreadyInProgressError when a second invocation arrives with the same payload before the first has finished, which is the in-progress branch made explicit. Its IAM requirements are dynamodb:GetItem, PutItem, UpdateItem and DeleteItem on the table.

Expiry, and what a stale key costs you

Idempotency records must expire or the table grows without bound. Powertools documents expires_after_seconds as defaulting to 3600, one hour. That is a sensible default for an HTTP retry and short for a queue, because the window you actually need to cover is the message’s maximum lifetime in the system: retention plus the maximum number of receives multiplied by the visibility timeout. On SQS, where retention defaults to 4 days and maxes at 14, an hour is comfortably too short.

Set the expiry above that total and use the store’s own TTL rather than a cleanup job. Too short and a legitimate late redelivery re-runs the model. Too long is merely storage, which is the cheaper mistake by several orders of magnitude — so err long.

A last point that surprises people: at the time of writing, the major model providers do not document an idempotency key on their completion endpoints the way payment APIs do. Provider SDK auto-retries are internal retries of a request whose response you never saw, and each one is capable of producing a billed generation. That is not a criticism of the SDKs, it is the reason this page exists: nothing upstream of your worker is going to deduplicate for you, so the check has to be yours. See what retries actually cost for the spend side of the same problem.

Building it

1. Generate the key at the point the work is first decided, and put it in the message body. If you hash, hash a documented canonical subset.
2. Create the store with the key as the primary key and TTL enabled on an expiry attribute. Set the TTL above retention plus maximum receives times visibility timeout.
3. Claim with a conditional write before the model call, recording in-progress and a lease. Never read-then-write.
4. Branch three ways on the result: proceed, return the stored result and ack, or release the message so it is retried later.
5. Update the record to done with the result attached in the same step that persists the answer, so the two cannot disagree.
6. Test it by deliberately redelivering: send the same message twice and confirm exactly one provider call appears in your request log. Not one result row — one call.