Testing an Agent's Behaviour When a Tool Call Times Out Mid-Loop

An agent asks for a tool, the tool is a database query behind a VPN that has just gone away, and the request never returns. What the user sees depends entirely on a code path that most agent loops have never exercised: what the loop does with a tool that neither succeeds nor fails, it simply does not answer.

Three ways a loop handles this, two of them wrong

• It hangs. No deadline on the tool call, so theawait waits as long as the underlying client does — which for a default HTTP client can be minutes or forever. The request holds a connection, the user sees a spinner, and the only thing that eventually resolves it is a proxy timeout somewhere else.
• It throws out of the loop. There is a deadline, and when it fires the exception propagates past the loop and terminates the run. The conversation is lost, including the model’s reasoning and every successful tool result before this one. From the user’s side one flaky dependency destroyed a task that was most of the way done.
• It reports the failure back to the model. The deadline fires, the loop appends a tool result marked as an error, and the model gets another turn. It can retry the tool, try a different tool, or tell the user it could not reach the system. This is the behaviour worth testing, and it is the only one of the three that treats the model as a participant in error handling rather than as a thing that broke.

The third is available in the provider APIs directly. In Anthropic’s Messages format a tool result block carries an is_error flag alongside its tool_use_id and content; in the OpenAI-shaped chat format the equivalent is a tool-role message whose content is your error text, correlated by tool_call_id. Either way the correlation id is mandatory: a tool result that does not name the call it answers is a malformed request, and dropping the block entirely is worse, because most APIs reject a turn that leaves a tool call unanswered.

What to assert

Not “an error was thrown”. The assertions that describe the behaviour you want are structural:

• The message list ends with a tool result whose id matches the timed out call, marked as an error. Assert the id match, not just the presence.
• The model was called again after the timeout — a call count of two, not one. This is what separates “handled” from “swallowed”.
• The loop terminated, and within its step budget. A recovery path that lets the model retry the same broken tool indefinitely is a new failure mode, not a fix.
• The error content contains no stack trace, no connection string and no internal hostname. It is going into a prompt, and from there possibly into an answer.
• The eventual, late resolution of the abandoned tool call does not append anything and does not produce an unhandled rejection.

What you must not assert is the model’s reply. “It should say it could not reach the database” is a test of prose, and it will fail on a model swap, a temperature change or a Tuesday. Assert that the model was given the chance and that the loop stayed well-formed; if you need to check the reply, check that it did not fabricate a result — for example that no fabricated row id appears — which is an assertion about absence and is stable.

One more assertion is worth adding and is easy to forget: that the deadline is per tool call and not per loop. A budget consumed by the whole run means a slow first tool leaves nothing for the second, and the symptom is a conversation that fails at a different point every time depending on which tool was slow — which reads as flakiness and is a design decision nobody made. Assert two sequential tool calls each get the full timeout.

Making a tool hang, deterministically

A setTimeout in the test is a race. Use a promise you control and never resolve until the assertions are done, and drive the deadline with fake timers so the test does not actually wait.

import { describe, it, expect, vi, afterEach } from "vitest";

function hangingTool() {
  let release!: (v: string) => void;
  const promise = new Promise<string>((resolve) => (release = resolve));
  return { promise, release };
}

// The tool runner under test.
export async function callTool(
  fn: (signal: AbortSignal) => Promise<string>,
  ms: number,
): Promise<{ ok: true; value: string } | { ok: false; error: string }> {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), ms);
  try {
    const value = await fn(controller.signal);
    return { ok: true, value };
  } catch {
    return { ok: false, error: `tool did not respond within ${ms}ms` };
  } finally {
    clearTimeout(timer);
  }
}

One detail here is load-bearing: callTool returns a result object rather than throwing. A tool runner that throws forces every caller to remember a try/catch, and the loop is exactly the place where forgetting one produces mode two above. Making the failure a value makes the recovery path unavoidable, and makes it something the type checker will remind you about.

The test

describe("agent loop with a timing-out tool", () => {
  afterEach(() => vi.useRealTimers());

  it("reports the timeout to the model and keeps the conversation", async () => {
    vi.useFakeTimers();
    const hang = hangingTool();

    const tools = {
      lookup_order: vi.fn(() => hang.promise),
    };

    const model = vi
      .fn()
      .mockResolvedValueOnce({
        stop_reason: "tool_use",
        content: [{ type: "tool_use", id: "toolu_01A", name: "lookup_order", input: { id: 9 } }],
      })
      .mockResolvedValueOnce({
        stop_reason: "end_turn",
        content: [{ type: "text", text: "I could not reach the order system." }],
      });

    const run = runAgent({ model, tools, toolTimeoutMs: 5_000, maxSteps: 6 });
    await vi.advanceTimersByTimeAsync(5_000);
    const result = await run;

    const last = result.messages.at(-1)!;
    const block = last.content[0];
    expect(block.type).toBe("tool_result");
    expect(block.tool_use_id).toBe("toolu_01A");
    expect(block.is_error).toBe(true);
    expect(block.content).not.toMatch(/at .*\.ts:\d+/);  // no stack trace

    expect(model).toHaveBeenCalledTimes(2);
    expect(result.steps).toBeLessThanOrEqual(6);
    expect(result.stopReason).toBe("end_turn");

    // The tool finally answers, long after nobody is listening.
    hang.release("order 9 found");
    await expect(run).resolves.toBe(result);
    expect(result.messages.at(-1)).toBe(last);
  });
});

The late result, and the budget

The final three lines are the part people leave out, and the bug they catch is real: a loop that keeps a reference to the abandoned promise and appends whatever it eventually returns will inject a tool result into a conversation that has already moved on, producing a message list the provider rejects as malformed — two results for one call, or a result with no preceding call. Aborting is not the same as cancelling; AbortSignal tells a cooperative client to stop, and an uncooperative one carries on and resolves later. The loop must decide once, at the deadline, and ignore the answer afterwards.

Add two more cases while the harness is in front of you. First, a tool that times out and then succeeds on the model’s retry, asserting the loop reaches a normal end_turn — recovery that only handles permanent failure is half-built. Second, a tool that times out every time, asserting the loop stops at its step budget with a distinguishable stop reason rather than looping until the context window fills. That second case connects to what tool results do to your context: each failed attempt appends a request and a result, so an unbounded retry is also an unbounded bill. Where a tool fails permanently rather than transiently, routing it out of the loop entirely is the subject of a dead-letter queue for tool-call failures.