Testing How Often a Model Calls a Tool It Was Not Given

The test is one line of logic: the tool name the model emitted must be a member of the set of names you sent. Everything difficult about this is deciding what happens when it is not, and which of the two tests you need runs on every commit.

The assertion is set membership

Do not assert on the model’s prose, and do not assert that a particular tool was chosen — both of those are tests of the model’s judgement and they will flake. Assert the one property that is not a matter of judgement at all: every tool call the loop receives names a tool that exists in the request.

const defined = new Set(tools.map((t) => t.name));
for (const call of toolCallsFrom(response)) {
  expect(defined).toContain(call.name);
}

That assertion has a property the interesting ones lack: it is a statement about your own request and your own dispatcher, so it is deterministic given a fixed response. It belongs in a unit test with a mocked transport, running in milliseconds, on every commit. The question of how often a live model does this is a different question with a different test, and mixing them gives you a slow suite that fails for two unrelated reasons.

Why a name that was never defined comes back

A tool call is generated text. On providers that constrain decoding to the tool schema, the function name is drawn from the names you supplied and an invented one is genuinely rare. The cases that still produce one are almost all structural rather than magical:

• The tool was removed but the transcript remembers it. You dropped search_invoices from the tools array last release. The conversation you are replaying still contains an assistant turn that called it and a tool result that answered it, and the model continues the pattern it can see. This is the most common cause and it does not require the model to invent anything.
• A sub-agent got a subset. The planner sees twelve tools, the worker is handed four, and the plan the worker is executing names one of the other eight.
• The name arrived in text, not in a tool call. The model wrote I will call get_balance now as prose, or emitted a JSON-looking block inside a text part. If your parser scrapes text for tool intent as a fallback, it will find names nothing constrained.
• A namespace collapsed. Tools registered from two MCP servers with a prefix applied in production and not in tests, so github__create_issue and create_issue are the same tool under two names and only one of them is defined in this request.

Test one: the dispatcher guard

The behaviour worth pinning is not that the unknown call never happens. It is that when it happens the loop survives it. An unknown tool name should produce a tool result marked as an error and fed back into the conversation, so the model gets a chance to pick a real tool, and it should increment a counter. It should not throw out of the loop, and it should not be silently dropped — a dropped call leaves the transcript with an assistant tool-use block and no matching result, which most providers reject on the next request.

import { describe, expect, it, vi } from "vitest";
import { runToolLoop } from "../src/agent";

const tools = [{ name: "get_balance", description: "...", input_schema: { /* ... */ } }];

describe("unknown tool names", () => {
  it("returns an error tool_result and keeps the loop alive", async () => {
    const transport = vi
      .fn()
      .mockResolvedValueOnce({
        stop_reason: "tool_use",
        content: [{ type: "tool_use", id: "tu_1", name: "get_invoices", input: {} }],
      })
      .mockResolvedValueOnce({
        stop_reason: "end_turn",
        content: [{ type: "text", text: "Your balance is 42." }],
      });

    const result = await runToolLoop({ tools, transport, input: "what is my balance" });

    // The second request must carry a tool_result for tu_1, marked as an error.
    const second = transport.mock.calls[1][0];
    const sent = second.messages.at(-1).content[0];
    expect(sent.type).toBe("tool_result");
    expect(sent.tool_use_id).toBe("tu_1");
    expect(sent.is_error).toBe(true);
    expect(result.metrics.unknownToolCalls).toBe(1);
  });
});

Three assertions, none of them about language. The pairing of tool_use_id to tool_result is the one people leave out, and it is the one that turns a handled error into a 400 on the very next request.

Test two: counting the rate against a real model

The rate is a monitoring question wearing a test’s clothes. Run it nightly, not per commit, over a fixed set of prompts that includes the awkward ones — a transcript containing a retired tool, a sub-agent prompt, a request where the obvious tool is deliberately absent. Record two numbers: the count of unknown names, and the set of distinct names that came back. The set matters more than the count. A single repeated name is a namespace or transcript bug you can fix today; a long tail of one-offs is model behaviour you have to absorb.

Report it as a threshold on a fixed denominator rather than as a pass or fail on a single run, because the run is sampled and a suite that fails on one sampled event will be muted within a fortnight. Fail the job when the count exceeds a number you chose deliberately, and print the distinct names either way.

The failure this actually catches

The bug this test finds is rarely a hallucination. It is drift between the tool catalogue your code registers and the tool catalogue your prompt describes. Somebody renames a tool, updates the registry, and leaves the system prompt saying “use lookup_customer to find an account”. The model does exactly as instructed, the dispatcher does not recognise the name, and the symptom presents as a quality regression rather than as a config error. Asserting membership turns that into a failing test with the offending name printed in it.

There is a second-order behaviour worth asserting once you have the guard. A model that receives an error result for an invented tool will often try the same name again, sometimes with slightly different arguments, and a loop that faithfully returns an error every time will burn its whole step budget on a tool that does not exist. So pin the escalation as well as the recovery: after the second unknown name in one turn, stop feeding errors back and end the turn with a message the caller can act on. That is one more mocked response in the same test and it is the difference between a handled error and an expensive one — see testing that an agent respects its step budget.

Keep the guard at the edge of any tool loop you did not write yourself, including one whose caller is another model. The registry is the boundary, and what a tool call actually is is worth re-reading if the idea that a tool name is generated text still feels surprising.