Testing a Model Swap Doesn't Change Your Function-Calling Schema Contract

Swapping the model behind a tool-calling feature looks like a configuration change and is not. The tool schemas are unchanged, your parsing code is unchanged, and the shape of what arrives can differ in three specific ways — each of which fails somewhere other than where it was caused.

What a swap actually breaks

The argument encoding. In the OpenAI-shaped API a tool call’s arguments arrive as a JSON string that you parse; in Anthropic’s Messages API the equivalent block carries an input object already parsed. Code written against one and pointed at the other either double-parses or treats an object as a string. If a gateway or SDK normalises this for you, the swap is invisible — and you should have a test proving that, rather than a belief.

Schema enforcement. Whether the provider guarantees the arguments validate against your schema, or merely encourages it, varies by model and by feature flag. Where a strict mode exists it usually supports a subset of JSON Schema, and a schema that was accepted by one model can be rejected by another for using a construct the strict subset excludes. A model without enforcement will occasionally return a plausible extra key, a number as a string, or an enum value adjacent to the ones you listed.

Tool selection. Given the same prompt and the same tools, models make different choices: calling two tools where one called one, calling none and answering directly, or picking a neighbouring tool when two descriptions overlap. This is the change most likely to be user-visible and least likely to be caught by a schema check, because a well-formed call to the wrong tool validates perfectly.

Structure yes, arguments mostly no

The assertion set that survives a swap and still catches breakage:

• The tool name is in the declared set. A hallucinated tool name is a real failure and a one-line check.
• Arguments validate against the schema you parse with. Use the same schema object the application uses, compiled once, so the test cannot drift from production. Set additionalProperties: false so an extra key fails rather than being silently dropped.
• Required fields are present and enums are in range. These come free from the schema, and they are where a weaker model fails first.
• Types are exact. A quantity arriving as "3" rather than 3 passes a truthiness check, passes a loose schema, and breaks arithmetic downstream.
• The correct tool was chosen, per case, where the case has one right answer. Assert the name, not the arguments.

Argument values are where restraint is needed. Assert them only where the case pins them: if the prompt says “refund order 88231”, then order_id must be 88231 and that is a fair, stable assertion. Do not assert a reason field, a summary, a generated title or any other free text — that is asserting on model prose, it will differ between models by design, and a test that demands identical wording from two different models is testing something nobody wants.

There is one more assertion worth having and it is about absence: that no tool call arrived for a case where none should. A model that is eager to call tools turns “what are your opening hours?” into a database query, which costs money, adds latency and can leak data into a context that did not need it. That is a genuine regression on a swap and it is invisible to any schema check, because the call that should not exist is perfectly well formed.

The case file

Keep the cases as data so the same list runs against every model, and include the negative cases, which are the ones that actually differ.

// fixtures/tool-cases.json
[
  {
    "name": "single tool, ids pinned by the prompt",
    "prompt": "Refund order 88231 for the full amount",
    "expectTools": ["issue_refund"],
    "expectArgs": { "order_id": "88231" }
  },
  {
    "name": "must not call a tool at all",
    "prompt": "What are your opening hours?",
    "expectTools": []
  },
  {
    "name": "ambiguous between two overlapping tools",
    "prompt": "Cancel my subscription",
    "expectToolsOneOf": [["cancel_subscription"], ["lookup_subscription", "cancel_subscription"]]
  },
  {
    "name": "enum must stay in range",
    "prompt": "Escalate this to the highest priority",
    "expectTools": ["escalate"],
    "expectArgs": { "priority": "urgent" }
  }
]

The third case is the honest way to handle non-determinism in tool selection: enumerate the acceptable sequences rather than pretending there is one. A model that first looks up the subscription and then cancels it is not wrong, and a test that fails it will be deleted.

The test

import { describe, it, expect } from "vitest";
import Ajv from "ajv";
import cases from "../fixtures/tool-cases.json";
import { TOOLS, callModel, normaliseToolCalls } from "../src/agent";

const ajv = new Ajv({ strict: false, allErrors: true });
const validators = Object.fromEntries(
  TOOLS.map((t) => [t.name, ajv.compile(t.input_schema)]),
);
const MODELS = [process.env.CURRENT_MODEL!, process.env.CANDIDATE_MODEL!];

describe.each(MODELS)("tool contract on %s", (model) => {
  it.each(cases)("$name", async (c) => {
    const raw = await callModel({ model, tools: TOOLS, prompt: c.prompt });
    const calls = normaliseToolCalls(raw);   // string args or object args, one shape out

    const names = calls.map((call) => call.name);
    if (c.expectToolsOneOf) {
      expect(c.expectToolsOneOf).toContainEqual(names);
    } else {
      expect(names).toEqual(c.expectTools);
    }

    for (const call of calls) {
      expect(Object.keys(validators)).toContain(call.name);
      const validate = validators[call.name];
      const ok = validate(call.arguments);
      expect(ok, JSON.stringify(validate.errors)).toBe(true);
      expect(typeof call.arguments).toBe("object");   // never a JSON string
    }

    for (const [key, value] of Object.entries(c.expectArgs ?? {})) {
      expect(calls[0].arguments[key]).toBe(value);    // exact, and only for pinned values
    }
  });
});

Running the same file across both models with describe.each is the point: the swap decision is then a question about a diff of two test runs rather than about intuition. Report the results as a matrix — case by model — so a candidate that passes everything except two ambiguous-selection cases is a visible, discussable outcome rather than a red build.

Strictness is not portable

One trap deserves naming. If your current model enforces the schema server-side, your application may have quietly stopped validating, because nothing invalid ever arrived. Swap to a model without that guarantee and the missing validation surfaces as a crash in whatever consumes the arguments — far from the swap, in code that has worked for a year.

So validate the arguments in your own code regardless of what the provider promises, and test that path: feed a deliberately invalid tool call through the normaliser and assert it is rejected cleanly — a defined error, a repair attempt, or a tool result marked as an error that gives the model another turn. That is the same recovery shape as a tool that times out mid-loop, and it is what makes a swap boring.

The related failure, where the model id does not change but its behaviour does, is a silent model update — and the same case file is the detector for it if you run it on a schedule rather than only at swap time.