Testing That a Translated Prompt Still Triggers the Same Tool Calls

Tool selection is a classification problem the model solves from the tool descriptions, which are almost always written in English, against a user turn that may not be. That asymmetry is where a multilingual agent breaks first, and it breaks silently: the model answers in prose instead of calling anything, which looks like a helpful response.

Why tool selection degrades before prose does

Producing fluent text in a language is something a model does across its whole vocabulary. Choosing between eight tools requires matching the user’s intent against eight short English descriptions and committing to one, and that match is a narrower, more brittle operation. It is also all-or-nothing: a slightly worse paraphrase is still a usable answer, whereas a slightly wrong tool choice runs the wrong code.

The failure that costs the most is not choosing the wrong tool. It is choosing no tool. A model that is less certain about the mapping falls back on what it is always able to do, which is answer in words — so the user asking in Turkish to cancel their subscription gets a polite explanation of how to cancel a subscription, and no cancellation happens. Nothing errors. The general mechanism is covered in why a tool call does not fire; this page is about proving it does not happen per language.

Name, arity, argument keys

Assert three things and stop. The tool name, the number of calls, and the set of argument keys. Everything else is either allowed to vary or is a different test.

• Name. Exact string match against the tool you registered. This is a contract you own, so exactness is correct here in a way it never is for prose.
• Arity. Assert the count, including asserting zero for cases that must not call a tool. A model that calls the right tool twice will double-charge a card, and a suite that checks only the first call will pass.
• Argument keys. The exact key set your schema defines, with no extras. Models occasionally add a plausible key, and strict schema validation is what catches it — see strict mode for structured outputs.

What you do not assert on is the assistant text accompanying the call. It is prose, it varies by language by design, and pinning it produces exactly the brittle test this whole cluster exists to argue against.

Which argument values may legitimately differ

Argument values sit in between, and deciding case by case is what makes the suite honest. Some must be identical across languages: an order id extracted from the text, a boolean flag, an enum drawn from your own schema. Some are allowed to differ and must be normalised before comparison:

• Place names. A weather tool called from a German prompt may receive "München" where the English one sends "Munich". Both are right. Compare through a normaliser, or assert membership in an accepted set held in the fixture.
• Dates. If the schema demands ISO 8601, the value is an invariant and any locale formatting is a bug. If the schema accepts free text, this is not a tool-call test, it is a schema problem — fix the schema.
• Free-text query arguments. A search string will be in the user’s language and should be. Assert it is non-empty and, if it matters, that it is in the input language; do not assert its content.
• Numbers. Assert the parsed numeric value with a tolerance, never the string. Decimal separators differ by locale and a string comparison will fail on a correct extraction.

Write the normalisation into the fixture rather than the assertion, so the rule about what is allowed to vary lives next to the data instead of being reinvented in each test.

Two response shapes, one assertion

Tool calls arrive differently depending on the provider. OpenAI’s chat completions place them in a tool_calls array on the message, each entry carrying a function object with name and a arguments string that you must parse as JSON yourself. Anthropic’s Messages API returns content blocks, and a tool call is a block with type of tool_use, carrying name and an already-parsed input object. Both are documented; neither is convertible to the other by accident.

Extract to your own shape once, in a helper, and let every test assert against that. Otherwise the test is coupled to a provider and adding the second one duplicates every case.

// tests/support/tool-calls.ts
export type ObservedCall = { name: string; args: Record<string, unknown> };

export function toolCalls(response: unknown): ObservedCall[] {
  const r = response as any;
  if (Array.isArray(r?.content)) {
    return r.content
      .filter((b: any) => b.type === "tool_use")
      .map((b: any) => ({ name: b.name, args: b.input }));
  }
  const calls = r?.choices?.[0]?.message?.tool_calls ?? [];
  return calls.map((c: any) => ({
    name: c.function.name,
    args: JSON.parse(c.function.arguments),
  }));
}

import { describe, expect, it } from "vitest";
import { toolCalls } from "./support/tool-calls";
import { agent } from "../src/agent";

const cases = {
  en: "Cancel my subscription please",
  de: "Bitte kündigen Sie mein Abonnement",
  tr: "Aboneliğimi iptal edin lütfen",
  ja: "サブスクリプションを解約してください",
};

describe.each(Object.entries(cases))("cancel intent in %s", (lang, text) => {
  it("calls cancel_subscription exactly once", async () => {
    const calls = toolCalls(await agent(text));
    expect(calls.map((c) => c.name)).toEqual(["cancel_subscription"]);
    expect(Object.keys(calls[0].args).sort()).toEqual(["confirm", "reason"]);
  });
});

The four causes of a language-specific miss

When one language fails, the cause is nearly always one of four, and they are distinguishable.

The descriptions are English-only. Most common, and worth testing directly: translate the tool descriptions too and rerun. If the failure clears, you have found it, and the fix is a decision about whether to ship localised descriptions — which costs tokens per request and means a description change now has to be made in every language.

An enum value is English. A reason parameter constrained to too_expensive | not_using | switching is fine, but if it is unconstrained free text the model will produce the user’s language and any downstream comparison against English breaks. Constrain the enum.

Truncation. Non-Latin scripts consume more tokens for the same content, so a prompt that fits in English may not fit elsewhere, and the tool definitions are often the part at the end that gets trimmed. Log the rendered prompt’s token count per language in the test output; a monotone increase across a failing set of languages names this instantly.

The model. Once the first three are excluded, this is genuinely a capability difference, and the answer is a routing rule rather than a prompt edit.