Testing That a Tool-Calling Agent Stops When the Task Is Done

The loop ends when the model stops asking for tools. That sounds like a single condition and it is three, only one of which means the work is finished — and the other two are the ones that produce a confidently truncated answer nobody notices.

What the completion signal actually is

The structural signal is an assistant turn with no tool calls in it. On Chat Completions that is an empty or absent tool_calls array with finish_reason of "stop". On the Messages API it is content with no tool_use block and stop_reason of "end_turn". Both are documented by their providers as the ordinary end of a turn, and both are what your loop should be reading.

The important half is the second condition. Checking only for the absence of tool calls conflates a finished task with a truncated one, because a response cut off at the token limit also has no tool calls in it — it has half a sentence. Read the reason as well as the shape.

It is worth being precise about where the check belongs. The stop condition is a property of the response you just received, so it is evaluated immediately after the call returns and before anything is appended to the message list. Loops that evaluate it at the top of the next iteration work, mostly, and pay for it with one wasted model call per conversation — which is the assertion in the test below and the reason the test exists at all.

type Ending =
  | { done: true; text: string }
  | { done: false; reason: "truncated" | "filtered" | "provider_error" };

function classify(res: any): Ending | null {
  const choice = res.choices[0];
  const calls = choice.message.tool_calls ?? [];
  if (calls.length > 0) return null;                 // keep looping
  switch (choice.finish_reason) {
    case "stop":           return { done: true, text: choice.message.content };
    case "length":         return { done: false, reason: "truncated" };
    case "content_filter": return { done: false, reason: "filtered" };
    default:               return { done: false, reason: "provider_error" };
  }
}

The three endings that look identical

• Finished. The model answered. This is the only case where returning the text to the caller is correct.
• Truncated. The response hit max_tokens. On Chat Completions finish_reason is "length"; on the Messages API stop_reason is "max_tokens". Treating this as done ships a half-written answer, and it is worse than it sounds because the half that survives reads as complete. Worse still, a response truncated during a tool call has no usable call in it either, so a loop that only counts tool calls silently drops the model’s intended action.
• Refused or filtered. The turn ended for a policy reason. There is no answer and there will not be one from a retry of the same request.

The distinction matters because the correct handling differs. A truncation can be retried with a higher limit or with an instruction to be briefer. A filter cannot. And a finished task should not be retried at all. A loop that maps all three to “done” makes every one of those decisions wrong in the same direction.

There is a fourth ending worth naming even though it is not a stop reason: the model returns a turn with neither tool calls nor useful text, because it emitted only whitespace or an empty string. This happens rarely and it is indistinguishable from completion by the structural check alone, so treat empty content with a terminal reason as a failure rather than as an answer. Returning an empty string to a caller who asked a question is the one outcome that is never correct, and a single assertion that the returned text is non-empty on the complete path costs nothing.

Testing the stop, and the extra lap

Two assertions carry this test. That the loop returned the right status for each ending, and that it made exactly the number of model calls the script contains — no extra lap after the final turn.

import { describe, it, expect, vi } from "vitest";

const stopTurn = (text: string) => ({
  choices: [{ finish_reason: "stop",
    message: { role: "assistant", content: text } }],
});
const truncatedTurn = () => ({
  choices: [{ finish_reason: "length",
    message: { role: "assistant", content: "The refund has been" } }],
});

it("stops on the finishing turn without calling again", async () => {
  const search = vi.fn(async () => ({ hits: ["a"] }));
  const model = vi.fn()
    .mockResolvedValueOnce(toolCallResponse("search", { q: "x" }, "c1"))
    .mockResolvedValueOnce(stopTurn("Found one result."));

  const res = await runAgent("find x", { model, tools: { search } }, 8);

  expect(res.status).toBe("complete");
  expect(res.steps).toBe(2);
  expect(model).toHaveBeenCalledTimes(2);   // not 3
  expect(search).toHaveBeenCalledTimes(1);
});

it("does not report a truncated turn as complete", async () => {
  const search = vi.fn();
  const model = vi.fn().mockResolvedValueOnce(truncatedTurn());

  const res = await runAgent("find x", { model, tools: { search } }, 8);

  expect(res.status).not.toBe("complete");
  expect(res.status).toBe("truncated");
});

toHaveBeenCalledTimes(2) is the assertion that catches the expensive bug. A loop that evaluates its stop condition at the top of the next iteration rather than immediately after the response makes one extra model call on every single conversation — invisible in output, fully billed, and adding a whole round trip of latency to every request you serve.

The truncation test is short because there is nothing to script: one turn in, one classification out. Add its sibling for a response truncated mid-tool-call, where finish_reason is "length" and the partial arguments string will not parse. That case belongs to the parse layer, but the loop has to route it there rather than treating the turn as finished.

Run both tests against a limit far above the number of scripted turns. If the limit is two and the script is two turns long, a loop that never detects completion still passes, because the guard stops it at exactly the right count for the wrong reason. Set the limit to eight, script two turns, and the call-count assertion can only be satisfied by a stop condition that actually fired.

Why not to assert on a done marker in prose

The tempting shortcut is to instruct the model to end with TASK_COMPLETE and have the loop look for that string. It works in a demo and fails in three predictable ways. The model emits it in the middle of an explanation of what it is about to do. The model translates it, or wraps it in backticks, or writes TASK COMPLETED. Or it stops emitting it after a prompt edit nobody connected to the loop, and the agent runs to the step limit on every request while producing correct answers — the most expensive silent failure in this whole cluster.

A test built on the marker inherits every one of those, and worse, it passes: your fake emits the marker because you wrote the fake. Assert on the structural signal, which the provider generates and you cannot accidentally fake into agreement.

When an explicit finish tool is worth it

There is one legitimate version of the explicit signal, and it is not a string in prose: a real tool named something like submit_answer, declared in the tool list with a schema for the answer. The model finishes by calling it, and the loop stops on that tool name.

This is worth the extra tool when the answer needs structure — a decision plus a confidence plus a citation list — because you get schema validation on the final output for free, and the assertions that come with a schema are far stronger than anything you can assert about prose. The cost is that the model can now end a turn without calling it, so you need both conditions: stop on submit_answer, and also handle the plain terminal turn, because a model that answers in prose without calling the tool must not be pushed round the loop again. Test both paths, and test that neither is reached before the step limit fires — which is the guard on the other side of this.