Skip to content
Multigrid

Testing tool calls and retrieval

Asserting on which tool ran, what it was given, when the loop stopped, and whether the retriever still returns what it did.

The hard part of testing a model-driven system is not that the output is random. It is that the interesting behaviour is not in the output at all. When an agent picks the wrong tool, loops forever, passes a negative quantity to a refund endpoint, or quietly stops retrieving the one document that made the answer correct, the prose it returns can look fine. A test that reads the prose finds none of it.

Everything a model does that you can assert on cleanly is structural: a tool name, an argument object, the number of times the loop went round, the ids that came back from the index, the order they came back in. These pages are about that layer. Each one takes a single failure — a hallucinated function name, a guard that does not fire, a chunking change that splits a paragraph in the wrong place — and shows the assertion that catches it, with the fake-model plumbing kept as small as it can be while still running.

Testing That the Model Picked the Right Tool for a Given Input

A small labelled set of input-to-expected-tool pairs, run as a fast structural check rather than as part of a scored eval suite.

9 min read

Testing an Infinite Loop Guard in a Multi-Step Tool-Calling Agent

Force a fake model to request the same tool forever and prove the step limit stops it, with the test itself bounded so a broken guard fails fast.

9 min read

Testing Tool Argument Validation Before Execution

Asserting that malformed, out-of-range or hostile arguments are rejected at the dispatch boundary and never reach the real tool function.

10 min read

Mocking a Tool's Return Value to Test the Next Model Turn

Fixing what a tool returns so a test can reach a specific multi-turn state, and asserting on the request the model receives rather than on its reply.

9 min read

Testing What Happens When the Model Calls a Tool That Doesn't Exist

The crash from a hallucinated function name, why the obvious fix produces a provider 400, and what the loop should return to the model instead.

9 min read

Testing a Multi-Turn Tool-Calling Conversation End to End

A scripted conversation fixture that drives the whole loop across several turns, with assertions on the transcript rather than on the final sentence.

10 min read

Testing That a Tool-Calling Agent Stops When the Task Is Done

Asserting on the completion signal — an empty tool-call list and a terminal finish reason — and on the truncation that looks exactly like it.

9 min read

Testing Parallel Tool Calls in a Single Turn

Asserting on a set of tool calls rather than a sequence, and on the message shape each provider requires when several results come back at once.

10 min read

Fixture Design for Recorded Tool-Calling Conversations

A JSON conversation-log format that stays readable in a diff, survives a provider change, and can be reused across several test cases.

9 min read

Unit Testing a Retriever With a Fixed Fake Corpus

A corpus small enough that the right document is unarguable, with assertions on rank rather than on similarity scores.

10 min read

Mocking the Vector Store to Test Generation Logic in Isolation

Fixing the retrieved context so the test can assert on the prompt your code assembles, including the cases a real index rarely produces.

10 min read

Writing a Regression Test for a Chunking Change

A fixture of query-to-required-span pairs that proves a chunking change has not split a document where an answer used to live.

10 min read

Testing That a Retriever Returns the Same Documents After a Reindex

Comparing ranked results before and after a rebuild with an overlap measure and a tolerance band, rather than asserting exact equality.

10 min read

Snapshot Testing Retrieved Chunks Before They Reach the Prompt

Committing a readable artefact of what retrieval returned, so a retrieval regression shows up in a diff instead of inside a generated answer.

10 min read

Other topics