Testing Webhook Delivery for an Asynchronous LLM Job

A webhook is an at-least-once delivery channel operated by somebody else. Every property worth testing follows from that sentence, and none of them is “does the callback fire”.

What you are actually testing

Long-running generation is increasingly offered as a job: you submit, you get an id, and a callback arrives when it finishes. OpenAI’s webhooks documentation describes the delivery contract concretely — requests are signed with webhook-id, webhook-timestamp and webhook-signature headers; the SDK exposes an unwrap method on its webhooks client that verifies and parses in one step; a completed background response arrives as an event of type response.completed; and delivery is retried with exponential backoff for up to 72 hours if the endpoint does not return a 2xx quickly, with 3xx redirects treated as failures rather than followed. See OpenAI’s webhooks guide for the current details.

Read that contract as four requirements on your endpoint. It must reject anything unsigned or misdated. It must tolerate the same delivery arriving more than once. It must return 2xx faster than the sender’s patience. And it must not be the only way your system learns a job finished, because 72 hours of retries ends in giving up.

Raw bytes and the signature

Signature verification runs over the exact bytes that were sent. The bug that catches nearly everyone is a framework that parses JSON before the handler sees it, so verification runs over a re-serialised copy with different key order, different unicode escaping or different whitespace, and fails on a genuine request. It is a configuration bug, not a crypto bug, and it is invisible until a payload happens to re-serialise differently.

Test it with a payload designed to be re-serialisation sensitive: include a non-ASCII string, a key ordering that a serialiser would normalise, and a number that round-trips imperfectly. Then assert verification succeeds. That fixture is worth more than a dozen happy-path tests.

1. Valid signature over an awkward body. Sign the raw bytes, post them, assert 2xx and that the job record was updated.
2. Tampered body, original signature. Change one byte. Assert a 4xx and, crucially, assert that no side effect occurred — no record written, no downstream call made. Test the absence, not just the status code.
3. Stale timestamp. Sign correctly with a timestamp well outside your tolerance and assert rejection. This is the replay defence and it is usually the untested one.
4. Missing headers entirely. An unsigned request must not fall through to a permissive branch. Assert rejection with no secret configured too, so a misconfigured environment fails closed rather than accepting everything.

Idempotency on the delivery id

At-least-once means duplicates are normal, not exceptional. A retry after your endpoint timed out delivers a message you already processed fully; the sender never saw the 2xx. So the handler needs a key it can deduplicate on, and the delivery id header is that key.

Test it directly: post the identical signed request twice and assert that exactly one record exists, one downstream call was made, and both requests returned 2xx. That last part matters — the second delivery must be acknowledged, not rejected, or the sender keeps retrying a message you have already handled.

it("processes a redelivered webhook exactly once", async () => {
  const body = signedFixture("response.completed", { responseId: "resp_123" });

  const first = await post("/webhooks/llm", body);
  const second = await post("/webhooks/llm", body);   // same webhook-id

  expect(first.status).toBe(200);
  expect(second.status).toBe(200);
  expect(await countJobRecords("resp_123")).toBe(1);
  expect(downstream).toHaveBeenCalledTimes(1);
});

Then write the harder case: two deliveries arriving concurrently. If deduplication is a read followed by a write, both requests read “not seen” and both process. The fix is a unique constraint on the delivery id and treating the constraint violation as success; the test is to fire both without awaiting the first and assert the same single record. This is the same reasoning as what a retried request costs you applied at the receiving end.

Acknowledge fast, work later

The sender’s patience is measured in seconds, and processing an LLM job result — parsing, validating, writing, notifying — can exceed it. An endpoint that does the work before responding gets retried mid-work, which produces exactly the duplicate-processing problem the previous section fixes, plus a load multiplier.

So the handler should verify, record the delivery, enqueue, and return. The assertion is about ordering: with the queue’s consumer paused, post a webhook and assert the response is 2xx and the job is enqueued while no processing has happened. That test fails immediately if somebody later moves work into the handler for convenience, which is the whole point — the regression is invisible in production until a slow day.

The delivery that never arrives

Retries end. An endpoint that was down for a long deploy, a signing secret rotated without updating the sender, a firewall change: each produces jobs that completed upstream and were never recorded downstream, and no amount of webhook handler testing detects it because the handler was never called.

The answer is a reconciler, and it deserves the test that closes this page. Poll for jobs your database believes are still in flight past a reasonable deadline, fetch their status directly, and complete them by the same code path the webhook uses. Two assertions: a job completed only via reconciliation ends in exactly the same state as one completed via webhook, and a job that gets both a late webhook and a reconciliation is still processed once. The second is the same idempotency key doing its job from a second direction, which is a good sign the design is right.