Parallel Tool Calls and When They Break Things

When a model returns three tool calls in one message, it has expressed an opinion about independence, not issued an instruction about scheduling. Whether they may actually run at the same time is your decision, and the model is not qualified to make it.

What the model is actually asking for

The tool_calls field is an array. A model that wants to read three files can emit three entries in one assistant message, and you get all three in a single response. Nothing in the protocol says they are concurrent. You may run them sequentially, concurrently, or refuse some — the only obligation is what you send back.

The reason models do this is latency: one round trip instead of three, and three round trips through a loop is three full re-sends of a growing context. The saving is real and worth having. It is also the reason the feature is enabled by default in most SDKs, which is how teams end up with concurrency they never designed.

The protocol obligation

One rule, and it is absolute: every tool_call_id in the assistant message must have a corresponding tool message before the next request. Not most of them. Not the successful ones. All of them, including the ones that errored, the ones you refused, and the ones you cancelled because a sibling failed first.

Miss one and the next request fails validation with the same class of error described in the agent loop — the conversation is structurally invalid, and the failure surfaces one step later than the bug, which makes it unpleasant to find. For a refused call, send a tool result saying so: REFUSED: send_email was requested alongside a write; serialised, not executed. Re-request it alone if still needed. The model reads that and re-requests. Silence produces a 400.

The four hazards

• Read-after-write within a step. The model emits write_file(a) and read_file(a) together, having assumed an order it never stated. Run them concurrently and the read may precede the write. This is the most common one, and it is silent — you get stale content, not an error.
• Non-idempotent duplicates. Two calls to send_invoice for the same customer in one step, because the model reasoned about it twice. Sequential execution does not help here; deduplication and idempotency keys do.
• Shared mutable state. Tools that implicitly depend on a working directory, a database transaction, a browser tab or a selected environment. Concurrency makes the implicit state a race. If two tools can touch one cd, they are not parallel-safe however innocent they look.
• Fan-out against a rate limit. Eight concurrent search calls hit a per-key limit, six return 429, and the model reads six failures as “search is broken” and abandons the approach. Bound your own concurrency below the provider’s limit; a semaphore is cheaper than a replan.

A three-class test

Tag every tool at definition time. This is a property of the tool, not of the request, so it belongs next to the schema:

The rule that falls out is a single sentence: within one step, run all pure and reads-mutable calls concurrently, then run at most one mutating call, and refuse the rest. It is conservative, it costs a round trip in the rare case, and it removes all four hazards above except duplication.

Duplication needs the other half: an idempotency key derived from the call itself rather than from a random value. key = sha256(run_id + tool_name + canonical_json(args)) means the same logical action requested twice in one run collapses to one effect, whether the duplicate came from the model, from your retry logic, or from a resumed run after a crash. Every serious payments API takes such a header for exactly this reason; agent tools need it more, not less.

A concurrent executor that refuses

from concurrent.futures import ThreadPoolExecutor

CLASS = {"search_docs": "pure", "read_file": "reads-mutable",
         "list_dir": "reads-mutable", "write_file": "mutating",
         "send_email": "mutating"}
MAX_CONCURRENCY = 4

def execute_step(calls, dispatch):
    """Returns tool messages in the same order as the calls arrived."""
    results = [None] * len(calls)
    concurrent, mutating = [], []
    for i, c in enumerate(calls):
        kind = CLASS.get(c["function"]["name"], "mutating")   # unknown = unsafe
        (mutating if kind == "mutating" else concurrent).append((i, c))

    # phase 1: every read, together
    with ThreadPoolExecutor(max_workers=MAX_CONCURRENCY) as pool:
        futures = {pool.submit(dispatch, c): i for i, c in concurrent}
        for fut, i in futures.items():
            results[i] = fut.result()

    # phase 2: one write, then refuse the rest
    for n, (i, c) in enumerate(mutating):
        if n == 0:
            results[i] = dispatch(c)
        else:
            results[i] = ("REFUSED: only one mutating tool runs per step. "
                          "This call was not executed. Re-request it in the "
                          "next step if it is still what you want.")

    return [{"role": "tool", "tool_call_id": c["id"],
             "name": c["function"]["name"], "content": str(results[i])[:8000]}
            for i, c in enumerate(calls)]

Three things worth copying even if you write your own. Unknown tools default to mutating, so forgetting to classify a new tool fails closed. Results are collected by index so the tool messages come back in the model’s original order, which keeps the transcript stable and therefore cacheable. And the refusal is a normal tool result with instructions in it — the model handles it as information, which is the general principle behind agent error recovery.

One property worth preserving while you are in there: determinism of the transcript. Two runs that make the same calls should produce byte-identical message lists, which means never ordering results by completion time and always formatting a given tool’s output the same way. It costs nothing, and it is what makes a prompt cache hit, an eval reproducible, and a diff between two traces readable.

One last option people forget: you can turn the whole thing off. Setting parallel_tool_calls: false where the provider supports it forces one call per message, costs you round trips, and eliminates this entire class of bug. For an agent whose tools mostly mutate things, that is not a cop-out — it is the correct default, and you can relax it per phase once you have the classification table above.