A Dead-Letter Queue Test for Tool-Call Failures, Not Message Failures

Standard dead-letter queue advice assumes a message that cannot be processed: bad payload, missing field, consumer bug. An agent introduces a second, different failure — the message was perfectly fine and a tool the model chose to call did not work — and putting both in one queue makes neither actionable.

Two failures that look alike

A message failure is a defect in the work item. The payload will not deserialise, references a deleted entity, or hits a consumer bug that reproduces on every attempt. Retrying is pointless; the message goes to a DLQ after its redelivery limit and a human looks at it. Nothing about this is specific to agents and it is thoroughly covered ground.

A tool failure is a defect in the world the agent is acting on. The run is valid, the model made a reasonable choice, and the tool returned a 500, timed out, hit its own rate limit, or refused because a downstream record was locked. The critical difference is that the model can often handle it: told the tool failed, it can retry it, use a different tool, or finish without it. That makes the first response in-band — a tool result marked as an error, back into the loop, as in handling a tool timeout mid-loop.

Only when in-band recovery is exhausted does it become a queue problem, and even then it is a different problem: the run is partially complete, expensive work has already been paid for, and the useful action is to resume from the failing step rather than to restart. A shared DLQ loses that distinction — the consumer cannot tell “replay this message from the beginning” from “retry step 4 of this run”, and the on-call rota for a malformed payload is not the rota for a payments API being down.

The routing rule

State it explicitly, because ambiguity here is what produces the shared queue:

• Cannot parse or validate the message → message DLQ, immediately, no retries. It will never succeed.
• Tool failed transiently, budget remains → not a queue event at all. Error tool result, model gets another turn.
• Tool failed and in-band recovery is exhausted → tool DLQ, carrying the run state.
• Tool failed because it is misconfigured for every run — auth rejected, endpoint gone → tool DLQ, and open the circuit so the next thousand runs do not each discover it individually. See circuit breakers.
• The consumer crashed mid-run → neither. The message is redelivered and the run resumes; this is the case that needs idempotency, not a DLQ.

The fourth and fifth are where implementations go wrong most often. A misconfigured tool sends every run to the DLQ within minutes, which is technically correct routing and operationally a flood — the breaker is what turns ten thousand DLQ entries into one alert. And a consumer crash routed to a DLQ loses runs that would have completed fine on redelivery.

What a tool-failure entry must carry

A message DLQ entry is the original message. A tool DLQ entry cannot be, because the original message no longer describes the state: the run is partway through. It must carry enough to resume.

type ToolFailureEntry = {
  runId: string;
  stepIndex: number;             // which turn of the loop failed
  toolName: string;
  toolArgs: unknown;             // redacted per the same rules as your logs
  attempts: number;              // in-band retries already spent
  lastError: { kind: string; status: number | null; message: string };
  conversationRef: string;       // pointer to stored messages, not the messages
  tokensSpent: { input: number; output: number };
  firstSeenAt: string;
  originalMessageId: string;
};

Two fields carry most of the value. conversationRef is a pointer rather than the conversation itself: message histories are large and often contain user content, and a DLQ is a place things accumulate for weeks with looser access controls than your primary store. And tokensSpent is what makes the replay decision rational — resuming a run that has already burned 40,000 tokens is worth more than restarting it, and without the number nobody knows which they are choosing.

The test

Three cases, one per destination, asserting the routing rather than the failure. Fake both queues as arrays; the thing under test is which one receives the entry.

import { describe, it, expect, vi, beforeEach } from "vitest";
import { handleMessage } from "../src/worker";

let messageDlq: any[];
let toolDlq: any[];
let queues: { messageDlq: typeof messageDlq; toolDlq: typeof toolDlq };

beforeEach(() => {
  messageDlq = [];
  toolDlq = [];
  queues = { messageDlq, toolDlq };
});

describe("dead-letter routing", () => {
  it("sends a malformed message to the message DLQ and never starts a run", async () => {
    const model = vi.fn();
    await handleMessage({ body: "{not json" }, { model, tools: {}, queues });

    expect(messageDlq).toHaveLength(1);
    expect(toolDlq).toHaveLength(0);
    expect(model).not.toHaveBeenCalled();     // no tokens spent on a broken payload
  });

  it("keeps a transient tool failure in-band and out of both queues", async () => {
    const lookup = vi.fn()
      .mockRejectedValueOnce(new ToolError({ kind: "server", status: 503 }))
      .mockResolvedValueOnce({ order: 9 });

    const result = await handleMessage(validMessage, {
      model: modelCallingLookupThenAnswering(),
      tools: { lookup }, queues, toolRetryBudget: 2,
    });

    expect(lookup).toHaveBeenCalledTimes(2);
    expect(result.status).toBe("completed");
    expect(messageDlq).toHaveLength(0);
    expect(toolDlq).toHaveLength(0);
  });

  it("sends an exhausted tool failure to the tool DLQ with resumable state", async () => {
    const lookup = vi.fn().mockRejectedValue(new ToolError({ kind: "server", status: 503 }));

    const result = await handleMessage(validMessage, {
      model: modelAlwaysCallingLookup(), tools: { lookup }, queues, toolRetryBudget: 2,
    });

    expect(messageDlq).toHaveLength(0);
    expect(toolDlq).toHaveLength(1);

    const entry = toolDlq[0];
    expect(entry.toolName).toBe("lookup");
    expect(entry.attempts).toBe(3);
    expect(entry.stepIndex).toBeGreaterThan(0);
    expect(entry.conversationRef).toMatch(/^conv_/);
    expect(entry.tokensSpent.input).toBeGreaterThan(0);
    expect(JSON.stringify(entry)).not.toContain(CUSTOMER_EMAIL);   // redaction applies here too
    expect(result.status).toBe("tool_failed");
  });
});

The negative assertions are as important as the positive ones. Every case asserts the queue it should not reach, because the failure mode being tested is misrouting rather than loss — an entry in the wrong queue is not missing, it is being handled by the wrong process, and only a cross-assertion catches that. The redaction check belongs here too: a DLQ payload is written by a path that usually skips whatever sanitisation your logger applies.

Testing the replay

A DLQ that nobody can drain is an expensive log. The replay path needs its own tests, and they are the ones most often missing.

1. Resume from the step, not the start. Replay an entry with a now-working tool and assert the model was called for the remaining steps only — count the calls. This is the assertion that proves the conversationRef is actually being used and not quietly ignored in favour of a fresh run.
2. Replay is idempotent. Replay the same entry twice and assert any side effect happened once, keyed on the run id and step index. Draining a DLQ is exactly when somebody runs the script twice.
3. A permanently broken run is discarded cleanly. Assert that an entry whose conversation reference has expired fails with a distinguishable status rather than throwing, so a drain of a thousand entries does not stop on the first bad one.
4. The queue is bounded and observed. Assert the metric increments and that entries carry firstSeenAt, so “oldest entry age” is alertable. A DLQ without that metric is discovered during the next incident.

One design note worth recording alongside the tests: if a tool failure is genuinely common for a given tool, the DLQ is the wrong destination and the agent should have a documented fallback — a different tool, a degraded answer, a handoff. The DLQ is for the failures you did not plan for, and its entry count is a decent signal of which failures you should have. When a tool is failing for everyone rather than for one run, the answer is upstream — see diagnosing a tool call that does not fire.