Testing Parallel Tool Calls in a Single Turn

A model that asks for three tools in one turn is not doing three turns quickly. It is one response with three calls in it, and the loop has to run all three, produce a result for each, and send them back in a shape the API accepts — before the model sees any of them.

One turn, several calls

Both APIs express this as multiple entries in one response. Chat Completions puts several objects in tool_calls on the single assistant message, still with finish_reason: "tool_calls". The Messages API puts several tool_use blocks in the same content array, still with stop_reason: "tool_use".

The loop bug this catches is the one written by someone who has only ever seen a single call: tool_calls[0]. It runs the first tool, produces one result, and sends a request in which two calls went unanswered — which the API rejects, so at least the failure is loud. The quieter variant runs all the tools but appends the results in a shape only one provider accepts.

function parallelResponse(calls: { tool: string; args: object; call: string }[]) {
  return {
    choices: [{
      finish_reason: "tool_calls",
      message: {
        role: "assistant",
        content: null,
        tool_calls: calls.map((c) => ({
          id: c.call,
          type: "function",
          function: { name: c.tool, arguments: JSON.stringify(c.args) },
        })),
      },
    }],
  };
}

Assert on a set, not an array

Which order the model lists the calls in is not something you control and not something worth pinning. The same prompt can produce [weather, traffic] on one run and [traffic, weather] on the next, and a test asserting the array order fails for a reason that has nothing to do with correctness.

import { describe, it, expect, vi } from "vitest";

it("runs every tool the model asked for, once each", async () => {
  const get_weather = vi.fn(async () => ({ c: 15 }));
  const get_traffic = vi.fn(async () => ({ delay_min: 4 }));
  const get_events  = vi.fn(async () => ({ events: [] }));

  const { model, requests } = player([
    parallelResponse([
      { tool: "get_weather", args: { city: "Lisbon" }, call: "c1" },
      { tool: "get_traffic", args: { city: "Lisbon" }, call: "c2" },
      { tool: "get_events",  args: { city: "Lisbon" }, call: "c3" },
    ]),
    stopTurn("Mild, light traffic, nothing on."),
  ]);

  await runAgent("what's Lisbon like right now?",
    { model, tools: { get_weather, get_traffic, get_events } });

  for (const t of [get_weather, get_traffic, get_events]) {
    expect(t).toHaveBeenCalledTimes(1);
    expect(t).toHaveBeenCalledWith({ city: "Lisbon" });
  }

  const second = requests[1];
  const resultIds = second.filter((m: any) => m.role === "tool")
    .map((m: any) => m.tool_call_id);
  expect(new Set(resultIds)).toEqual(new Set(["c1", "c2", "c3"]));
  expect(resultIds).toHaveLength(3);          // set equality hides duplicates
});

The length assertion after the set comparison is not redundant. new Set(["c1","c1","c2","c3"]) equals new Set(["c1","c2","c3"]), so a loop that duplicates a result passes the set check alone. Duplicated results are a real failure mode — they come from a dispatcher that appends inside a nested loop — and the API will reject them.

Where the calls are independent, running them concurrently is the point: three sequential tool calls at 200ms each cost 600ms, and Promise.all costs 200. Assert that too, not with a timing measurement, but by having each stub record the moment it started and checking the last start precedes the first completion. Timing assertions in CI are flaky; ordering assertions are not.

The result shape is not the same on both APIs

This is the part that breaks a loop ported between providers, and it is worth an explicit test on each shape you support.

Chat Completions wants n separate messages, each with role: "tool" and its own tool_call_id, all appended after the single assistant message. The Messages API wants one user message whose content array holds n tool_result blocks, each with its tool_use_id — as shown in Anthropic’s tool use documentation. Emitting three user messages there, one per result, is invalid.

it("batches results into one user message on the Messages shape", () => {
  const built = buildToolResults([
    { call: "tu_1", output: { c: 15 } },
    { call: "tu_2", output: { delay_min: 4 } },
  ], "messages");

  expect(built).toHaveLength(1);
  expect(built[0].role).toBe("user");
  expect(built[0].content.map((b: any) => b.type))
    .toEqual(["tool_result", "tool_result"]);
  expect(built[0].content.map((b: any) => b.tool_use_id))
    .toEqual(["tu_1", "tu_2"]);
});

Testing the builder directly rather than through the loop is deliberate: it is a pure function from a list of results to a list of messages, which makes it the cheapest thing in the system to test exhaustively across both shapes and every arity including zero.

The zero case is not hypothetical. A turn where the model requested no tools should produce no result messages at all, and a builder that returns one empty user message there sends a message with an empty content array, which both APIs reject. It is a one-line test and it fails on more implementations than you would expect.

When one of them fails

Partial failure is the case that separates a loop that works from one that works in production. Two of three tools succeed and the third throws. The wrong answers are to abandon the turn, or to send back only the two that worked; both leave a call unanswered.

The right behaviour is that every call gets a result, and the failed one gets a result describing the failure. On the Messages API set is_error: true on that block. On Chat Completions there is no such flag, so encode it in the content. Then assert the successful results are unaffected — a common bug is a Promise.all that rejects on the first failure and discards the other two results that had already completed, which turns one flaky tool into a wasted turn for all three. Promise.allSettled is the fix, and the assertion is that all three ids appear even when one stub rejects.

Test the timeout case the same way. A tool that never returns must produce a result saying so, within your deadline, or the parallel batch is as slow as its worst member and the loop cannot proceed at all.

Testing with parallelism turned off

Both providers let you suppress this. Chat Completions takes parallel_tool_calls: false; the Messages API takes disable_parallel_tool_use: true inside the tool_choice object. If your production configuration sets either, your tests must set it too — a suite exercising parallel batches against a system configured for one call per turn is testing a path that never runs.

It is worth keeping the parallel tests even if you disable the feature, marked as the behaviour you would need if it were turned back on. Disabling parallelism costs a full round trip per additional tool, so the pressure to re-enable it arrives with the first latency complaint, and by then nobody remembers whether the loop ever handled it.

One last case belongs in this file rather than elsewhere: the same tool requested twice in one turn with different arguments — two weather lookups for two cities. It is legal, it happens, and it breaks any dispatcher that keys results by tool name instead of by call id. Assert that both calls ran, that each got its own arguments, and that the two results are attached to the correct ids. A loop that pairs them the wrong way round produces an answer that is confidently, precisely backwards, which no assertion on the prose would ever catch.