Why an Agent's Tool-Choice Behaviour Changes After a Provider Swap

The tool definitions did not change. The prompt did not change. The agent now calls search_docs where it used to call get_order_status, or calls nothing and answers from memory. Four mechanical causes produce exactly this before any question about the model arises — check them, then treat what remains as a measurable difference rather than a defect.

The symptom

Selection failures come in four flavours and it is worth naming which one you have before diagnosing, because they have different causes: the agent picks a plausible but wrong tool; it picks no tool and answers directly; it calls a tool that does not exist; or it calls the right tool with the wrong arguments. Only the first two are selection problems. The third is usually a definition or dispatch problem, and the fourth is a schema problem.

Rule out the mechanical causes

• The definitions did change, in translation. The conversion between provider schemas can drop a field that carries meaning — most often a per-property description, or an enum that a converter did not carry across. Serialise the outbound tools array on both providers and diff them as text. Descriptions are the model’s only guide to what a tool is for, so losing one is losing the selection signal itself; the design of those descriptions is covered in tool description design.
• The system instruction moved or vanished. If routing guidance lives in the system prompt — “always check order status before searching documentation” — and the system prompt was dropped in the message-shape translation, selection loses its rules. This is the same failure that quietly wrecks eval scores; see migrating an evaluation harness.
• Tool order changed. Definitions are an ordered array, and a converter that iterates a dictionary can reorder them. Position is not supposed to matter and in practice it is not free. Fix the order deterministically so it is not a variable.
• tool_choice did not survive. Forcing values are spelled differently on each side, and an unrecognised value can be ignored rather than rejected — so an agent you believed was forced to call a tool is now choosing freely. The values and their equivalents are in mapping the tool_choice parameter between APIs.

If the diff is clean and tool_choice is faithful, you have a genuine behavioural difference. That is the interesting case.

Why selection is model-specific

Tool selection is not a lookup. The model is producing a continuation conditioned on the tool descriptions, the conversation, and whatever it learned during training about when calling a function is appropriate. That last part is the variable a provider swap changes, and it changes it in ways that are systematic rather than random:

• Propensity to call at all. Models differ in how readily they reach for a tool versus answering from parametric knowledge. A model with a lower propensity will answer “your order is probably still processing” where the previous one looked it up — fluent, confident and unsourced.
• How the description is weighted against the name. Where a tool’s name suggests one thing and its description says something narrower, models differ in which they follow. Tools whose names are near-synonyms of each other are where this shows up first.
• Behaviour with overlapping tools. If two tools could plausibly serve a request, the choice is a judgement, and different training produces a different judgement. Overlap you had never noticed becomes visible on the new model.
• Multi-step planning. Whether the model calls one tool, sees the result and then calls the next, or emits several calls at once, differs — and an agent whose second step depended on the first result behaves differently when both are requested up front.

Note what this is not: it is not one model being better at tool use. Two models can have equal accuracy on your task and different selection distributions, and the one that matches your prompt’s assumptions will look better until you rewrite the prompt for the other.

Measuring selection instead of arguing about it

Do not settle this by trying a few prompts by hand. Build a fixed set of requests with a known correct tool per request — a hundred drawn from real traffic beats a thousand invented — and run it against both providers with everything else pinned. Then report three numbers per model:

selection accuracy  = correct tool chosen / total
no-call rate        = answered without any tool / total
confusion matrix    = chosen tool x correct tool

The confusion matrix is the one that tells you what to do, because selection errors are not uniform: they concentrate in two or three pairs of tools whose descriptions overlap, and those pairs are the fix list. An aggregate accuracy number hides that entirely.

Keep this as a regression suite rather than a one-off, since the same drift happens on a model version bump without any migration at all — testing correct tool selection and silent model updates cover that ground.

What actually shifts selection

1. Disambiguate the confused pairs first. Rewrite the two descriptions so each states what it is for and when to prefer the other: “Use for order status by id. For questions about policy or how something works, use search_docs instead.” Cross-references between tool descriptions are unusually effective and cost a handful of tokens.
2. Remove or merge genuinely overlapping tools. If two tools have similar descriptions because they do similar things, no amount of prompting will make the choice reliable. The best fix for a selection problem is often fewer tools.
3. Raise the propensity to call, if the problem is no-call. A system instruction that says the model must not answer factual questions about customer data from memory addresses the actual failure. Where a call is genuinely mandatory, force it with tool_choice rather than asking.
4. Constrain the tool set per turn. Passing only the tools relevant to the current phase removes the confusion instead of resolving it, and it reduces the schema tokens you pay for on every turn — see tool schema token cost.
5. Re-run the fixed set after every change and check the confusion matrix, not the headline number. A prompt change that fixes one pair and breaks another is a common outcome and only the matrix shows it.