Shimming Parallel Tool Calls on a Provider That Only Allows One

Your agent loop assumes an assistant turn can contain three tool calls, runs them concurrently, and appends three results. The new provider emits one per turn. The shim keeps the array shape your loop expects and fills it one call at a time — and that is a different conversation than the one you had before, in a way worth understanding before you ship it.

What parallel tool calls actually are

The name suggests concurrency in the model. There is none. What actually happens is that a single assistant turn emits more than one tool invocation before yielding, and the concurrency is entirely yours: you execute the invocations however you like, then return all of the results in the next turn, and the model continues from there. In OpenAI’s Chat Completions shape the assistant message carries a tool_calls array and you reply with one message of role tool per call, each carrying the matching tool_call_id (OpenAI, Chat Completions reference). In Anthropic’s Messages shape the assistant turn carries several tool_use content blocks and you reply with a single user turn containing the corresponding tool_result blocks (Anthropic, Messages API). Both APIs also expose a way to turn the behaviour off, which is worth knowing because you may be looking at a gap you configured.

So the property you lose when a provider emits one call per turn is not speed of execution. It is that the model committed to a set of calls without seeing any of their results. That independence is the whole semantic content of “parallel” here, and it is the thing the shim cannot reproduce.

The serialising loop

The structure is straightforward. Keep asking the model for one call, execute it, append the result, ask again — and stop when the model stops asking or you hit a cap. The subtlety is in what you hand back to the caller and when.

type ToolCall = { id: string; name: string; args: unknown };

// Emulates "give me every tool call for this turn" on a provider that
// returns at most one. Returns the same array shape a parallel-capable
// provider would have returned in a single response.
async function turnWithSerialisedCalls(
  client, messages, tools, { maxCalls = 6 } = {},
) {
  const executed: { call: ToolCall; result: string }[] = [];
  const working = [...messages];

  for (let i = 0; i < maxCalls; i++) {
    const res = await client.complete({ messages: working, tools });

    const call = firstToolCall(res);          // null if the model answered
    if (!call) {
      return { text: textOf(res), calls: executed.map(e => e.call) };
    }

    const result = await runTool(call);       // your dispatcher
    executed.push({ call, result });

    // Append in the target provider's own history shape. This is the part
    // that differs per API; see the tool-call mapping page.
    working.push(assistantToolCallMessage(call));
    working.push(toolResultMessage(call, result));
  }

  // Cap reached. This is a real outcome, not an exception to swallow.
  throw new ToolCallBudgetExceeded(executed.map(e => e.call));
}

Two details in there are load-bearing. First, working is a copy: the shim appends several turns of its own to the history, and if it mutates the caller’s array those synthetic turns leak into the next request even when the caller intended to discard them. Second, the budget exhaustion is thrown rather than returned as a successful answer. A model that keeps asking for tools until the cap is a model that is not converging, and returning empty text as though it had finished is how that becomes a silent product bug.

If the provider offers no tool support at all, this is not the page you need — that is a much larger shim in which you describe the tools in the prompt and parse invocations out of prose, with all the fragility of the structured-output shim and none of the schema help.

Keeping the caller’s interface

The point of the shim is that the code above it does not change. That means three things must be synthesised faithfully.

• Identifiers. If the provider does not supply a call id, generate a stable one and use it in both the invocation record and the result record. Correlating results to calls by tool name works until the same tool is called twice, which is the exact case a parallel-call shim exists to serve. Google’s Gemini API keys function responses by name rather than by an id (Google, generateContent reference), which is a real source of ambiguity when a tool is invoked more than once in a turn.
• Ordering. Return the calls in the order they were actually made. A caller that logs them, or replays them into an evaluation, will treat the order as meaningful, and here it genuinely is — unlike in a real parallel turn, call two depended on call one.
• Error shape. A failed tool must be reported to the model in the shape the provider expects, not as a thrown exception that ends the loop. Anthropic’s tool_result block carries an is_error flag; the OpenAI-shaped equivalent is a tool message whose content is a description of the failure, with no dedicated flag. Either way the model gets a turn to react, and an agent that cannot see its own tool failures cannot recover from them.

What serialising changes

This is the section to read before deciding the shim is finished.

In a parallel turn, the model chooses all of the calls from the same state of knowledge. If it asks for the weather in three cities, it asks for all three because it has decided it needs all three. In the serialised version, it asks for the first city, sees the answer, and then decides what to ask for next. Usually it asks for the second city. Sometimes it does not: having learned that the first city is warm, it may conclude the question is answered, or it may pursue a detail in the first result rather than continuing the sweep. The conversation is no longer the same conversation, and the difference is not a bug in your loop.

The consequences are concrete. Coverage becomes unreliable for tasks where you were implicitly relying on the model to enumerate: a three-city sweep can come back with two. Results are no longer independent, so a tool that returns something misleading now contaminates every subsequent choice in the turn rather than sitting alongside the others. And any evaluation built against the parallel behaviour will drift, because it was measuring a property the shim does not preserve — testing parallel tool calls is worth revisiting when you turn this on.

There is one mitigation and it is not free: when your application knows the fan-out up front — the three cities are in your code, not in the model’s head — do not ask the model to enumerate at all. Call the tool three times yourself and put all three results in the context before asking anything. That removes the model from the enumeration entirely, which is more reliable than either the parallel call or the shim, and it is available far more often than people expect.

What it costs, and the caps you need

A parallel turn with three calls is two model requests: one to get the calls, one to continue after the results. Serialised, the same work is four requests. The extra cost is not evenly distributed. Each request re-sends the whole conversation, which by the fourth iteration includes all previous invocations and all previous results, so the input token count grows with each step and tool results are often the largest thing in the context. Latency compounds the same way: you pay time-to-first-token once per step rather than once, and the steps are strictly sequential.

That makes three caps mandatory rather than defensive. A maximum call count per turn, which the code above has. A token budget for the working conversation, checked before each iteration, because the growth is superlinear in the number of steps. And a wall-clock deadline for the whole turn, because a serialised loop behind a user-facing request can exceed a request timeout without any single call being slow. This library’s test that a tool loop stops at its cost budget is the regression worth having before this ships.

If prompt caching is available on the route, this is where it earns its keep: the stable prefix — system prompt and tool schemas — is re-sent on every iteration, and a serialised loop re-sends it several times per turn instead of twice. Whether it applies depends on how the provider decides a prefix matches, and appending to the end of the conversation is exactly the pattern caching handles best.

Build it

1. Confirm the gap rather than assuming it. Send one request with two obviously independent tools and a prompt that needs both, and count the invocations in the response. Check also that you have not disabled parallel calls yourself through a configuration flag.
2. Decide whether the fan-out is knowable in your own code. If it is, call the tools yourself and skip the loop — that is the better answer and it is available more often than not.
3. Write the loop over a copy of the message array, appending one invocation turn and one result turn per iteration in the target provider’s history shape.
4. Synthesise a stable id per call if the provider does not give one, and use it on both the invocation and the result so repeated calls to the same tool stay distinguishable.
5. Return tool failures to the model as results, using the provider’s error convention, instead of throwing out of the loop.
6. Add all three caps: call count, working-context tokens, wall-clock deadline. Make budget exhaustion a distinct outcome the caller can see.
7. Return the executed calls in order, in the same array shape a parallel-capable provider would have returned, so nothing upstream changes.
8. Re-run whatever tests asserted enumeration coverage. Expect at least one to fail, and treat that failure as information about the shim rather than as a flake to retry.