Testing Tool Argument Validation Before Execution

The model does not call your function. It emits a name and a bag of arguments, and something you wrote decides whether to invoke anything. That decision point is the only place tool arguments can be checked, and it is testable with no model in the loop at all.

Two layers, two different failures

On the Chat Completions shape, function.arguments is a string containing JSON. On the Messages API, input on a tool_use block is an already-parsed object. That difference means the first shape has a failure mode the second does not: arguments that are not valid JSON at all, usually because the response was cut off at max_tokens mid-object.

So there are two layers. Parse, which can fail with a SyntaxError and produces no object to inspect. Then validate, which gets an object and asks whether it is the right one. Conflating them produces the confusing report where a truncated response is filed as a schema violation, and the fix — raising max_tokens — is nowhere near where you are looking.

Both layers are worth testing even if you ship only one shape today, because the parsed-object shape carries its own trap: an object that arrived already parsed has passed no check whatsoever, and it is easy to write a dispatcher that validates the string path carefully and trusts the object path completely. The provider parsed the JSON. It did not verify that the values mean anything.

type Parsed =
  | { ok: true; args: unknown }
  | { ok: false; kind: "unparseable"; raw: string };

function parseArgs(raw: string): Parsed {
  try {
    return { ok: true, args: JSON.parse(raw) };
  } catch {
    return { ok: false, kind: "unparseable", raw };
  }
}

Validate at the dispatcher, not in the tool

The temptation is to let the tool function validate its own input, since it already knows what it needs. The reason not to is that the tool is also called from your own code, where the arguments came from a typed caller and are already trusted. Put the check in the dispatcher and it applies to exactly the untrusted path, once, for every tool.

import { z } from "zod";

const schemas = {
  create_refund: z.object({
    order_id: z.string().regex(/^[0-9]{5,8}$/),
    amount_cents: z.number().int().positive().max(50_000),
    reason: z.enum(["damaged", "late", "wrong_item"]),
  }).strict(),
};

type Dispatch =
  | { ok: true; result: unknown }
  | { ok: false; kind: "unparseable" | "invalid_arguments" | "unknown_tool";
      detail: string };

export async function dispatch(
  name: string,
  rawArguments: string,
  tools: Record<string, (a: any) => Promise<unknown>>,
): Promise<Dispatch> {
  const schema = schemas[name as keyof typeof schemas];
  if (!schema || !tools[name]) {
    return { ok: false, kind: "unknown_tool", detail: name };
  }
  const parsed = parseArgs(rawArguments);
  if (!parsed.ok) {
    return { ok: false, kind: "unparseable", detail: parsed.raw.slice(0, 200) };
  }
  const checked = schema.safeParse(parsed.args);
  if (!checked.success) {
    return {
      ok: false,
      kind: "invalid_arguments",
      detail: checked.error.issues
        .map((i) => `${i.path.join(".")}: ${i.message}`)
        .join("; "),
    };
  }
  return { ok: true, result: await tools[name](checked.data) };
}

Two schema choices are doing real work here. .strict() makes an unexpected property a failure rather than something silently dropped, which is how an invented currency field gets noticed instead of being ignored into a wrong-currency refund. And .max(50_000) is a range the schema knows about, so “refund forty thousand euros” is a validation failure rather than a business-logic surprise. In Python the same split is a Pydantic model with model_validate raising ValidationError, and extra="forbid" in the model config playing the part of .strict().

The cases worth writing

Not one per field. One per class of thing the model does wrong, which is a much shorter and more interesting list:

• Truncated JSON. A string ending mid-object. This is a real production event, not a hypothetical, and it is the one case that never reaches your schema at all.
• Right shape, wrong type. amount_cents as "4000" rather than 4000. Models produce string-typed numbers regularly, and a permissive coercion here is how a comparison against a numeric limit silently starts comparing strings.
• In range for the type, out of range for the world. A negative quantity, a refund larger than the order, a date in 1970.
• Invented enum member. reason as "customer_unhappy", which is plausible, absent from the enum, and exactly what a model produces when the schema description does not list the members.
• Missing required field. Especially where a model might reasonably have asked the user instead.
• Injection through an argument. A path with ../ in it, or a query string carrying instructions. The schema is not a security boundary on its own, but a pattern-constrained id field is where a traversal attempt stops being possible. See the defences that belong around it.

The assertion is that nothing ran

Asserting on the error text is the weak version of this test: the message changes when you upgrade the validator, and a passing string comparison tells you nothing about whether the refund went through. The assertion that matters is that the spy for the real tool was never invoked.

import { describe, it, expect, vi } from "vitest";

describe("argument validation", () => {
  const bad: [string, string][] = [
    ["truncated", '{"order_id":"55219","amount_c'],
    ["string number", '{"order_id":"55219","amount_cents":"4000","reason":"late"}'],
    ["negative", '{"order_id":"55219","amount_cents":-100,"reason":"late"}'],
    ["over cap", '{"order_id":"55219","amount_cents":900000,"reason":"late"}'],
    ["bad enum", '{"order_id":"55219","amount_cents":100,"reason":"unhappy"}'],
    ["extra field", '{"order_id":"55219","amount_cents":100,"reason":"late","currency":"GBP"}'],
    ["traversal", '{"order_id":"../../etc","amount_cents":100,"reason":"late"}'],
  ];

  it.each(bad)("rejects %s without calling the tool", async (_label, raw) => {
    const create_refund = vi.fn();
    const out = await dispatch("create_refund", raw, { create_refund });

    expect(out.ok).toBe(false);
    expect(create_refund).not.toHaveBeenCalled();
  });

  it("passes a valid call through unchanged", async () => {
    const create_refund = vi.fn(async () => ({ refund_id: "rf_1" }));
    const raw = '{"order_id":"55219","amount_cents":4000,"reason":"damaged"}';

    await dispatch("create_refund", raw, { create_refund });

    expect(create_refund).toHaveBeenCalledTimes(1);
    expect(create_refund).toHaveBeenCalledWith({
      order_id: "55219", amount_cents: 4000, reason: "damaged",
    });
  });
});

The happy-path case is not padding. Without it, a dispatcher that rejects everything passes every negative test, and toHaveBeenCalledWith on the exact object is what catches a validator that quietly coerces or renames on the way through.

One further assertion earns its place on the dangerous tools: that the rejection is recorded. A refund attempt for nine thousand euros that the schema stopped is not a non-event — it is either a model behaving badly or a user probing, and both want to be visible. Assert that the dispatcher called your audit hook with the tool name and the offending argument, and you have turned a silent guard into something you can count. What belongs in that record is a question of its own, but the rejected value and the constraint it broke are the minimum.

What you hand back to the model

Rejection is not the end of the turn. The model is waiting for a result for that call id, and both APIs require one — an unanswered tool call makes the next request invalid. So the validation failure becomes a tool result, and what you put in it decides whether the model can recover.

Return the field path and the constraint, not a stack trace and not a bare “invalid arguments”. On the Messages API set is_error: true on the tool_result block so the failure is marked as such; on Chat Completions there is no flag, so the role: "tool" message content carries it. Then assert the retry: feed the loop a fake model that emits a bad call first and a good one second, and check the tool ran exactly once, with the corrected arguments. Cap the retries — two attempts and then a hard failure — because a model that cannot satisfy the schema will not satisfy it on the fifth try either, and that is its own loop to guard.