Tool-Use and Function-Calling Benchmarks

10 min read · updated August 4, 2026

A function-calling benchmark has to answer a question with no obvious answer: what counts as the right call? The three families of answer — compare the parsed call to a reference, execute it and compare results, or run a whole conversation and compare the final database state — measure different things and produce numbers that are not interchangeable.

What these benchmarks test

The task is: given a set of tool definitions and a user request, emit the correct call or calls. That decomposes into several sub-skills that a single score merges.

Selection. Choosing the right function from the available set. Gets harder as the set grows — the relationship is charted in how many tools is too many.
Argument extraction. Pulling the parameter values out of the request, in the right types, with the right units and formats.
Schema conformance. Emitting syntactically valid output that matches the declared schema. Increasingly handled by constrained decoding rather than by the model — see JSON mode versus structured outputs.
Abstention. Not calling a function when none applies. This is a distinct skill and it is the one most likely to be missing.
Sequencing. Calling several functions in the right order, feeding one result into the next, across multiple turns.

Three ways to grade a call

Mechanism	Description
AST comparison	Parse the emitted call into a syntax tree and compare it structurally to a reference: same function name, all required parameters present, values matching the expected value or one of a permitted set, types correct. Nothing is executed. Cheap, deterministic, offline-runnable — and it rejects a correct call expressed differently from the reference.
Execution	Actually run the call against a real or simulated API and compare the result to the expected result. Catches calls that parse but fail, and permits alternative correct formulations that produce the same outcome. Depends on the external service being up and unchanged, which makes historical runs irreproducible.
Final-state comparison	Run a whole multi-turn conversation in a simulated environment with a database, then compare the resulting database state to an annotated goal state. Grades the outcome rather than the calls, so any sequence that achieves the goal passes. The most realistic and the most expensive.

The Berkeley Function Calling Leaderboard is the best-known example of the first two, running most categories by AST comparison and a subset by real execution against live APIs. The tau-bench family from Sierra is the clearest example of the third, simulating a retail or airline customer-service domain with a policy document, a simulated user played by a model, and a database whose end state is the grading target.

AST comparison has a specific and important weakness: it needs a reference call, and a reference call encodes one way of doing the thing. A model that calls a different-but-valid function, or passes an optional parameter the reference omits, can be marked wrong. Benchmarks mitigate this with permitted-value sets and multiple accepted references, and the mitigation is never complete.

The categories, and the one people forget

These suites decompose the task into categories rather than reporting a single number, and the decomposition is the useful part.

Category	Description
Simple	One function available, one call expected. The floor case, and where nearly every model is now strong.
Multiple	Several functions available, one call expected. Tests selection.
Parallel	One request that requires several independent calls at once. Tests whether the model emits a list rather than one call and a promise to do the rest later.
Parallel multiple	Several functions available and several calls needed. The combination of the two above.
Relevance / irrelevance detection	A request that no available function can serve. The correct behaviour is to call nothing and say so. This is the category that separates models, and the one omitted from most summaries.
Multi-turn / multi-step	A conversation where the result of one call informs the next, and the user may supply missing information partway through.

Irrelevance detection deserves its own paragraph because it is the production failure. A model that always calls something scores well on every category above except this one, and in a deployed agent it is the model that books a flight when the user asked a question about baggage policy. The over-eager tool caller and the over-eager answerer are the same failure of abstention, and the reasons it is hard to train for are in making a model say I do not know.

When you read a headline function-calling score, find the per-category table. A model at the top of the aggregate can be middling at irrelevance detection, and if your agent has a wide tool surface that is the number that will hurt you.

pass^k: consistency instead of coverage

Agentic tool benchmarks introduced a metric worth knowing because it inverts the familiar one. pass@k asks whether any of k attempts succeeded. pass^k asks whether all of k attempts succeeded.

Run each task n independent times; c of them succeed.

  pass@k  = 1 - C(n-c, k) / C(n, k)     "at least one of k works"
  pass^k  =     C(c,   k) / C(n, k)     "all k of k work"

Worked, n = 10 trials on one task:

  c = 8 successes:
    pass@1 = 8/10                = 0.800
    pass^1 = C(8,1)/C(10,1) = 8/10        = 0.800
    pass^2 = C(8,2)/C(10,2) = 28/45       = 0.622
    pass^5 = C(8,5)/C(10,5) = 56/252      = 0.222
    pass^8 = C(8,8)/C(10,8) = 1/45        = 0.022

  c = 5 successes:
    pass^2 = C(5,2)/C(10,2) = 10/45       = 0.222
    pass^5 = C(5,5)/C(10,5) = 1/252       = 0.004

The point of pass^k is that a customer-facing agent does not get to retry until it works. If eight runs in ten succeed, the chance that five consecutive customers all get a correct outcome is about one in five. A model that looks strong at pass@1 can be unusable at pass^5, and that collapse is exactly the reliability property production cares about.

Report pass^k for any agent that acts on the world. It is a considerably more honest number than a single-attempt accuracy, and it costs only the repeats you should be running anyway to estimate variance.

The gap between benchmark failures and production failures

A high function-calling score predicts less than it appears to, for a reason that is structural rather than a matter of benchmark quality: the failure distribution is different.

Where the mismatch is	Description
Tool count and quality	Benchmark tool sets are small, well-named and cleanly documented. Production tool sets grow organically, contain overlapping functions, inherited names and descriptions written under time pressure. Selection accuracy depends more on your schema quality than on the model.
Tool results	Benchmarks mostly supply clean results or none. Production tools return errors, empty sets, partial data, rate-limit responses and stale caches. Whether the model notices a tool failed is a large part of agent quality and is barely tested.
Prompt context	Benchmark prompts are short and clean. Yours arrives after a system prompt, several turns of history, and possibly retrieved documents, all competing for attention.
Constrained decoding	If you use a structured-output mode, schema conformance is enforced by the decoder and stops being a model property at all — so that portion of the benchmark score is measuring something you have already solved differently.
Consequences	A benchmark scores a wrong call as one point lost. In production a wrong call may write to a database. Blast radius is not in any score — gate placement is covered in human-in-the-loop design.

See where to put the approval gate and securing tool calls for the design responses. Neither is something a benchmark can score for you.

Testing your own tools

The good news about this category is that it is the easiest kind of evaluation to build yourself, because the grading is mechanical. You do not need a judge, human labels, or a reference answer in prose — you need your own schemas and a list of requests with expected calls.

Take fifty real user requests from your logs, including ones where no tool applies. A quarter of the set should be the abstention case.
Annotate each with the expected function name and the required arguments, allowing a set of permitted values where more than one answer is right.
Grade by parsing the emitted call and comparing structurally — the AST approach. Do not execute during evaluation unless your tools are read-only.
Run every item at least five times and report pass^5 as well as pass@1. Tool selection is more variable than people expect.
Break the score out by category: selection, arguments, abstention. Fixing a selection problem and fixing an argument problem are different pieces of work, usually both in the schema descriptions rather than in the model.