Testing That a Retried Async LLM Job Doesn't Fire Its Webhook Twice

A long-running generation job finishes, the worker posts a completion webhook, and something dies before the queue is told the job is done. The queue does what an at-least-once queue does and hands the job to another worker. The customer gets two webhooks, their system creates two records, and the model call — the expensive part — may well have been paid for twice.

The window where the duplicate is born

Write out the worker’s steps and the bug becomes obvious, which is the point of writing them out:

1. Receive the job message from the queue.
2. Call the model. This takes 40 seconds and costs money.
3. Persist the result.
4. POST the completion webhook to the customer’s endpoint.
5. Acknowledge or delete the queue message.

Every gap between steps 2 and 5 is a window. A crash between 2 and 3 loses a paid-for generation. A crash between 3 and 4 leaves a finished job that never notified anybody. A crash between 4 and 5 — or, much more commonly, a visibility timeout expiring because step 2 took longer than the queue expected — produces the duplicate webhook. Nothing crashed at all in that last case: two workers simply ran the same job because the first one was too slow to say it was still alive.

The fix is not “retry less”. At-least-once delivery is the guarantee you have; a system that cannot tolerate a redelivery is broken regardless of how rare the redelivery is. The fix is that every step after the model call is idempotent under a key that survives the retry — and that is what the test asserts.

What to assert

The assertion is a count, keyed on an identity that does not change between attempts. Concretely, after driving the worker twice over the same job:

• Exactly one delivery per job_id. Not “one delivery” — count the deliveries whose body carries that job id, so the test still means something when the harness has other traffic in it.
• At most one model call. If the result was already persisted, the second attempt must load it, not regenerate it. This is the assertion that turns a correctness test into a cost test.
• The two attempts carry the same idempotency key. Assert the header value directly. A key derived from a timestamp, a UUID generated at attempt time, or the queue’s own message id is different on the retry and therefore useless — and that mistake passes a naive “we send an idempotency key” test.
• The queue message is gone at the end. Otherwise you have proven the worker is idempotent and not that it terminates.

Derive the key from the job, not from the attempt: the job id itself, or a hash of the job id and the event name if a job can emit more than one event. Send it as an Idempotency-Key header and record it in your own outbox with a unique constraint, so the second attempt loses the race in the database rather than in application code.

A harness that can crash on demand

You cannot test this by calling the worker twice, because calling it twice tests the happy path twice. You need the process to stop between the delivery and the acknowledgement. The controllable way to do that is to make the acknowledgement a seam and have the test throw from it on the first attempt.

// worker.ts
export type Deps = {
  model: (prompt: string) => Promise<string>;
  deliver: (url: string, body: unknown, key: string) => Promise<void>;
  ack: (receipt: string) => Promise<void>;
  store: Store;
};

export async function runJob(job: Job, deps: Deps) {
  const existing = await deps.store.getResult(job.id);
  const result = existing ?? (await deps.model(job.prompt));
  if (!existing) await deps.store.putResult(job.id, result);

  const key = `job:${job.id}:completed`;
  // Unique index on (idempotency_key). Returns false if a row already exists.
  const claimed = await deps.store.claimDelivery(key);
  if (claimed) {
    await deps.deliver(job.webhookUrl, { job_id: job.id, result }, key);
    await deps.store.markDelivered(key);
  }

  await deps.ack(job.receipt);
}

Note the ordering: claim, then deliver, then mark. Claiming before delivering means a crash mid-delivery leaves a claimed-but-unmarked row, which is a deliberate choice of at-most-once over at-least-once for the notification. If your product needs the opposite, deliver first and dedupe at the consumer — but decide it, and write the test for whichever you chose, rather than discovering it in an incident.

The test

import { describe, it, expect, vi } from "vitest";

describe("completion webhook under retry", () => {
  it("delivers once when the ack fails after delivery", async () => {
    const deliveries: Array<{ body: any; key: string }> = [];
    const model = vi.fn(async () => "generated text");
    const deliver = vi.fn(async (_url: string, body: any, key: string) => {
      deliveries.push({ body, key });
    });

    const store = new InMemoryStore();
    const job = makeJob("job_7f3a", { webhookUrl: "https://x.test/hook", receipt: "r1" });

    // Attempt 1: everything works until the acknowledgement.
    const ackFails = vi.fn(async () => {
      throw new Error("visibility timeout expired");
    });
    await expect(runJob(job, { model, deliver, ack: ackFails, store })).rejects.toThrow(
      "visibility timeout expired",
    );

    // Attempt 2: the queue redelivers the identical message.
    const ackOk = vi.fn(async () => {});
    await runJob(job, { model, deliver, ack: ackOk, store });

    const forThisJob = deliveries.filter((d) => d.body.job_id === "job_7f3a");
    expect(forThisJob).toHaveLength(1);
    expect(forThisJob[0].key).toBe("job:job_7f3a:completed");
    expect(model).toHaveBeenCalledTimes(1);
    expect(ackOk).toHaveBeenCalledTimes(1);
  });
});

Three variants are worth adding once this one passes, because each catches a different real bug. Crash between claim and delivery and assert that a recovery pass re-attempts it — the claimed-but-unmarked row must not be a permanently lost notification. Run two attempts concurrently with Promise.all rather than in sequence, which is the interleaving a visibility timeout actually produces and which an in-memory boolean check passes while a real unique index is what saves you. And assert the second attempt does not call the model even when the first attempt’s putResult is slow, since a result written after the retry starts is the case where the deduplication has to be the database.

The half you do not control

Your test proves your worker sends one webhook. It cannot prove the customer receives one, because networks retry, load balancers retry, and your own delivery layer will retry a 5xx — correctly. So document the key: state in the webhook reference that Idempotency-Key is stable per event and that consumers should treat a repeat as a no-op. A delivery guarantee that only one side knows about is not a guarantee.

The same shape recurs everywhere a model call has a side effect attached: a tool that charges a card, a job that writes a row, a message that lands on another queue. In each case the assertion is a count keyed on something derived from the work rather than from the attempt, and in each case the tempting shortcut — an in-process flag, a “have we sent this?” lookup that is not a unique constraint — passes the sequential test and loses the concurrent one. When a tool’s side effect fails permanently rather than transiently, the routing question that follows is a different one, and it is the subject of a dead-letter queue for tool-call failures. The retry policy that opens these windows in the first place is worth costing as well, since every redelivery is a billed request: see what retries cost.