Testing That the Model Picked the Right Tool for a Given Input

Tool selection is a classification problem hiding inside a generation problem. The model has n tools and one input, and either it reaches for the right one or it does not. That is a label you can write down, which makes it one of the very few things about a model you can test the ordinary way.

Why this is not part of the eval suite

A full evaluation run scores answer quality: is the summary faithful, is the tone right, did it cite the source. Those need a judge model or a human, they cost real money per run, and they take minutes. Tool selection needs neither. The comparison is string equality against a name you chose, so the whole check is a single cheap model call per row and a fast local comparison.

Keeping them apart buys you the thing that matters: this check can run on every commit that touches a tool description, and the eval suite cannot. Tool descriptions are the input to selection — the description is what the model routes on — so the edit most likely to break selection is a one-line wording change nobody thinks of as risky. A ninety-second check that runs on that diff catches it; a twenty-minute suite that runs nightly finds out after it has merged.

You can go further and cap the request at a small max_tokens. You are not reading the answer, only the tool call, and a model that has decided to call a tool emits the call before it emits much else. Set the cap high enough that the arguments are not truncated mid-JSON and no higher.

The labelled pairs

Keep the set in one file, as data, not spread through test bodies. It is the thing you will be editing when a new tool lands, and it wants to be diffable on its own.

// tests/fixtures/tool-selection.ts
export type SelectionCase = {
  ref: string;
  input: string;
  expect: string | null;   // tool name, or null for "answer directly"
  why: string;             // why this row exists; read when it fails
};

export const cases: SelectionCase[] = [
  { ref: "wx-plain", input: "What's the weather in Lisbon?",
    expect: "get_weather", why: "canonical" },
  { ref: "wx-past", input: "Was it raining in Lisbon last Tuesday?",
    expect: "get_weather_history", why: "past tense must not hit the live tool" },
  { ref: "ord-status", input: "Where is order 55219?",
    expect: "get_order", why: "order number in the text" },
  { ref: "ord-refund", input: "Order 55219 arrived smashed, I want my money back",
    expect: "start_refund", why: "complaint, not a lookup" },
  { ref: "chit", input: "thanks, that's all",
    expect: null, why: "no tool; guards against a tool-happy prompt" },
  { ref: "amb-cancel", input: "cancel it",
    expect: null, why: "ambiguous: must ask, not guess a target" },
];

The why field is not documentation for its own sake. When a row fails eight months from now, the question is always “is this row still right?”, and a row whose reason nobody recorded gets deleted rather than investigated.

Build the set from cases that have gone wrong rather than from cases you imagine. Near-misses between two similar tools are worth ten unambiguous ones: get_weather against get_weather_history, a lookup against a mutation, a read tool against the write tool that shares half its vocabulary.

The assertion, in both provider shapes

The one thing you assert is the name. Not the arguments — that is a separate check with a different failure mode — and certainly not any prose that came alongside.

import { describe, it, expect } from "vitest";
import { cases } from "./fixtures/tool-selection";

function selectedTool(response: any): string | null {
  // Chat Completions: tool calls hang off the assistant message.
  if (response.choices) {
    const calls = response.choices[0].message.tool_calls;
    return calls?.[0]?.function?.name ?? null;
  }
  // Messages API: tool_use blocks live in the content array.
  const block = response.content.find((b: any) => b.type === "tool_use");
  return block?.name ?? null;
}

describe("tool selection", () => {
  it.each(cases)("$ref: $input", async (c) => {
    const res = await callModel(c.input);           // your one-shot wrapper
    expect(selectedTool(res)).toBe(c.expect);
  });
});

Two details in selectedTool are load-bearing. It reads tool_calls[0] and content.find rather than assuming a single-element array, because a model that requests two tools at once is a different failure than a model that requested the wrong one, and you want it to read as such. And it returns null rather than throwing when there is no call, so a “did not call anything” row and a “called the wrong thing” row produce comparable output.

The rows that expect no tool at all

These are the rows most sets are missing, and they are the ones that catch over-eagerness. A prompt tuned to stop the model answering from memory will happily push it into calling a tool for “thanks”, and nothing in a set of positive examples notices.

Two kinds are worth having. Conversational filler, where any call is waste. And genuinely ambiguous input — “cancel it” with no referent — where the correct behaviour is to ask, and calling cancel_order with a guessed id is the expensive bug. Note that this is exactly where forcing a call hurts: setting tool_choice to require a tool removes the model’s ability to be right about these rows, so if your production path forces a call, your test path must force it too or you are testing a different system.

Ambiguous rows have a second use. When one of them starts failing, the usual cause is not the model but a system prompt that grew an instruction like “always take action rather than asking questions”. That instruction is invisible in a diff of the tool definitions and completely changes selection behaviour, so a failing amb-cancel row is often the only signal that somebody edited the preamble. Keep the system prompt in the same fixture the suite loads, not baked into the client, so the check runs against the prompt that actually ships.

One more row class is worth having: an input whose correct tool changed. When you split search into search_docs and search_tickets, the old inputs should now select the narrower tool, and updating those rows in the same commit as the split is what makes the suite a record of intent rather than a record of history. Rows nobody updates become rows nobody trusts.

Pass threshold, not all-green

Unless you pin sampling to greedy decoding, this suite is not deterministic, and a single flipped row on an ambiguous input will fail a build for no reason. The fix is to score the set rather than assert each row, and fail below a line you chose.

it("selects the right tool on the labelled set", async () => {
  const results = await Promise.all(
    cases.map(async (c) => ({ c, got: selectedTool(await callModel(c.input)) })),
  );
  const wrong = results.filter((r) => r.got !== r.c.expect);
  for (const r of wrong) {
    console.error(`${r.c.ref}: expected ${r.c.expect} got ${r.got} (${r.c.why})`);
  }
  expect(wrong.length / results.length).toBeLessThanOrEqual(0.05);
});

Print every miss before asserting, so one failure tells you which rows broke rather than only that the rate moved. Keep a second, much smaller set of rows that must be exactly right — the ones where the wrong tool spends money or deletes something — and assert those individually with no threshold at all. A 95% floor is fine for routing between two read tools and is not fine for the refund endpoint.

Pin what you can while you are here. Temperature at zero removes most of the variance, and where a provider exposes a seed, setting it removes more — though neither buys you exact determinism, because providers change weights behind a stable model name and batching affects reduction order in floating-point arithmetic. Treat the threshold as protection against the residue, not as a licence to leave sampling loose. If a row flips on repeated runs at temperature zero, that row is genuinely ambiguous and belongs in the ambiguous set rather than the exact set.

Record the miss rate over time rather than only asserting on it. A number drifting from 2% to 4% over six weeks while never crossing 5% is the interesting signal, and it is invisible to a suite that reports only pass or fail. Writing the rate to a file the CI job keeps is enough; the same argument applies to quality metrics generally, and a labelled selection set is the cheapest one you will own.