Load Testing Concurrent Tool-Calling Requests

A single chat completion is one request and one response. A tool-calling agent is a loop: the model returns tool calls, you execute them, you send the results back, and the model may ask again. Load-testing that as if it were one request produces numbers that are wrong by whatever the average round-trip count happens to be — and you do not know that number until you measure it.

The unit under test is the loop, not the call

Start by naming what a “request” means for your service. If your users send one message and wait for one answer, then one user-visible request is one loop, and everything that matters — latency SLO, cost, error budget — is defined over the loop. Locust’s statistics table will happily show you a healthy p95 on /v1/chat/completions while your users wait eleven seconds, because the eleven seconds is four of those calls plus three tool executions plus the gaps.

So the loop gets its own timing, fired as a synthetic request event, and each leg gets its own name underneath it. That way one run produces a percentile table you can read top-down: the loop, then the model calls, then each tool.

The second thing to fix before writing any code is the round-trip distribution. Pull it from production logs if you have them: count how many assistant messages with a tool_calls array occurred per conversation. If the distribution is “92% one round trip, 7% two, 1% five or more”, a test that always does exactly two round trips overstates your model spend by roughly a factor of two and understates your tail. The tail is where agent loops fail, because that is where the context has grown large enough to be slow and where a loop guard has to fire.

Time each leg separately

There are at least four different latencies inside one loop, and they respond to completely different fixes:

• Model call latency, per round trip. This grows with each round trip, because the conversation carries every previous tool result forward. Round trip three is reading a much longer prompt than round trip one; if you report one blended figure, that growth is invisible.
• Tool execution latency, per tool. Your own services. Under load these are the first thing to degrade, and they degrade in a way that looks like the model being slow if you are not measuring them apart.
• Serialisation and glue. Parsing arguments, validating them against the schema, re-encoding the results. Usually negligible, occasionally not, and free to measure.
• Total loop time. The only one your user experiences.

A locustfile that drives the loop

The pattern is: send the messages, inspect the response for tool calls, execute them, append the results with matching tool_call_id values, and go round again with a hard cap on iterations. Every leg is timed with environment.events.request.fire, whose documented keyword arguments are request_type, name, response_time, response_length, response, context and exception.

# locustfile.py
import json, time
from locust import HttpUser, task, between

TOOLS = [{
    "type": "function",
    "function": {
        "name": "lookup_order",
        "description": "Look up an order by its id.",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
        },
    },
}]

MAX_ROUND_TRIPS = 6

class AgentUser(HttpUser):
    wait_time = between(2, 6)

    def fire(self, name, ms, exc=None, size=0):
        self.environment.events.request.fire(
            request_type="AGENT", name=name, response_time=ms,
            response_length=size, response=None, context={}, exception=exc,
        )

    def call_model(self, messages, trip):
        with self.client.post(
            "/v1/chat/completions",
            json={"model": "your-model", "messages": messages,
                  "tools": TOOLS, "tool_choice": "auto"},
            name="/v1/chat/completions trip=" + str(min(trip, 3)),
            catch_response=True,
        ) as r:
            if r.status_code != 200:
                r.failure("HTTP " + str(r.status_code))
                return None
            r.success()
            return r.json()

    def run_tool(self, call):
        start = time.perf_counter()
        args = json.loads(call["function"]["arguments"])
        result = {"order_id": args.get("order_id"), "status": "shipped"}
        self.fire("tool:" + call["function"]["name"],
                  (time.perf_counter() - start) * 1000)
        return json.dumps(result)

    @task
    def conversation(self):
        loop_start = time.perf_counter()
        messages = [{"role": "user", "content": "Where is order A-4192?"}]
        trips = 0
        try:
            while trips < MAX_ROUND_TRIPS:
                trips += 1
                body = self.call_model(messages, trips)
                if body is None:
                    raise RuntimeError("model call failed")
                choice = body["choices"][0]
                messages.append(choice["message"])
                calls = choice["message"].get("tool_calls") or []
                if not calls:
                    break
                for call in calls:
                    messages.append({
                        "role": "tool",
                        "tool_call_id": call["id"],
                        "content": self.run_tool(call),
                    })
            else:
                raise RuntimeError("loop guard hit at " + str(MAX_ROUND_TRIPS))
        except Exception as exc:
            self.fire("agent loop", (time.perf_counter() - loop_start) * 1000, exc=exc)
            return
        self.fire("agent loop", (time.perf_counter() - loop_start) * 1000)
        self.fire("agent round trips", trips)

The last line is a small trick that pays for itself: firing the round-trip count as if it were a response time gives you percentiles over it in the same table. You can then read the p99 round-trip count straight off the report and see how often the loop guard is close to firing.

Why your concurrency number is wrong

Locust spawns a fixed number of users, each of which is doing one thing at a time. On a single-call test, a hundred users is approximately a hundred in-flight HTTP requests. On an agent loop it is not, and the difference is large enough to invalidate a capacity plan.

Work it through with labelled assumptions. Suppose a loop averages three model calls at 2 s each and three tool executions at 100 ms each, with a wait_time averaging 4 s. One user then spends 6.3 s per loop doing work and 4 s waiting, so a loop occupies 10.3 s of which 6 s — about 58% — is an in-flight model request. A hundred users therefore hold roughly 58 concurrent model requests, not a hundred. Now degrade the provider so each model call takes 6 s instead of 2 s: the loop becomes 22.3 s, of which 18 s is in flight, and the same hundred users now hold about 81. Concurrency rises as the provider slows, which is exactly the direction that turns a slowdown into a rate-limit event.

Those numbers are arithmetic from the assumptions stated in the paragraph, not a measurement. Substitute yours. The point that survives any substitution is that a closed-loop load generator does not hold concurrency constant, so if you need a specific number of simultaneous requests against the provider — because that is what your quota is denominated in — you must measure in-flight requests directly rather than infer them from the user count.

What to assert while the load is running

Latency is the obvious output and the least interesting one. Under concurrency, an agent loop fails in ways a single call cannot:

• Every tool call has a matching result. Assert that each tool_call_id you received appears in exactly one message you sent back. Under concurrency, a bug that mixes results between conversations shows up here and nowhere else.
• The loop terminates. Count how often the guard fires. Zero is the expected value; anything above zero under load and near zero at rest means the loop is being extended by degraded tool responses, not by the task.
• Called tools are tools you declared. A name outside your tools array is a real production failure mode — see when a tool call does not fire for the inverse case.
• Arguments validate against the schema. Validate every arguments object before executing, and count failures as a separate statistic. That count rising under load usually means you have started truncating context.

None of these assert on the model’s prose, which is the point. The loop is deterministic in structure even when it is non-deterministic in content, and the structure is what breaks under load.

One last piece of bookkeeping makes the run comparable to the next one: record total tokens per loop, not per call. An agent loop re-sends the whole conversation on every round trip, so a three-trip loop over a 2,000-token base prompt sends roughly 2,000 + 2,400 + 2,800 input tokens rather than 2,000 — the growth being the tool results accumulating. Summing per-call usage across the loop is the only figure that translates into a cost per user action, and it is the number that grows fastest when a change makes the loop take one more round trip than it used to.