Toolformer and Learned Tool Use

Toolformer: Language Models Can Teach Themselves to Use Tools (2023) answers a question that sounds circular: how do you build a training set for tool use without anyone annotating when a tool should have been used? The answer is a filtering criterion, and it is the sort of idea worth stealing regardless of whether you care about the paper.

The question it asks

Language models are bad at things a calculator is good at, and cannot know anything after their training cutoff. Giving them access to external tools is obviously desirable. The hard part is not calling the API — it is deciding when a call is warranted, which tool, with what arguments, and how to use the result.

The obvious approach is supervised: have humans annotate a corpus with the calls that should appear. That is expensive, and it bakes in annotators’ opinions about when a tool helps, which may not match where the model is actually weak. The paper’s alternative is to let the model’s own uncertainty decide.

The method

The pipeline has three phases and no human labelling of tool calls.

• Sample. Take a plain text corpus. Using a handful of in-context examples per tool, prompt the model to propose places in the text where an API call could be inserted, and what the call would be. This generates a large number of candidate annotations, most of them useless.
• Execute and filter. Actually call the APIs, get the results, and keep only the candidates that pass the criterion in the next section.
• Fine-tune. Fine-tune the model on the surviving corpus — the original text with the useful calls and their results inlined. The model that results inserts calls itself during generation, and the calls are interleaved with ordinary text rather than being a separate mode.

The filter is the idea

The filtering criterion is where the self-supervision comes from. For each candidate call, compare how well the model predicts the tokens that follow in three conditions: with the call and its result available, with the call but no result, and with nothing. Keep the call only if having the result reduces the loss on the subsequent tokens by more than a threshold.

In other words: a tool call is useful precisely when knowing its answer makes the continuation less surprising to the model. That is a measurable quantity, it requires no annotation, and it automatically targets the model’s own gaps rather than a human’s guess at them. If the model already knows the date, the calendar call does not reduce its loss and is discarded.

This is a good specimen of a paper whose contribution is a well-chosen objective rather than an architecture. When reading, it pays to ask of any method: what is the quantity being thresholded, and is it a proxy for the thing we want? Here the proxy is unusually tight, which is why the method works with a fairly small filtered corpus.

What it showed and what it did not

The tools used are simple and few — a calculator, a question-answering system, a search engine, a translation system and a calendar — and the paper reports improved zero-shot performance on downstream tasks relative to the unmodified base model, while retaining the model’s core language modelling ability. That last check matters: a method that improves task scores by damaging the base model is a different and worse result, and the paper looks for it.

The limitations are stated and they are substantive. Calls are not chained — the model does not use the output of one tool as the input to another. There is no interactive use, meaning the model does not react to a failed call by trying a different query. And the sampling phase is expensive relative to the amount of data that survives filtering.

Every one of those limitations names something that is now a routine expectation of an agent. Reading the limitations section of an influential paper a few years later is one of the more instructive exercises available: it shows you what was hard, and how much of the subsequent work is simply the removal of those specific constraints.

The road the field took instead

Toolformer teaches a model to embed calls in its own generation. What became standard is different in an important way: models are trained during post-training to emit structured calls against schemas supplied at request time, and the application executes them and returns results in a subsequent turn. The tool set is not fixed at training time at all.

That difference is the entire reason function calling became a product feature rather than a research technique. You do not need a fine-tuning run per tool; you describe your tool in the request and the model, which has been trained on the general skill of reading a schema and producing a conforming call, handles one it has never seen. The mechanics of that interface are covered in how tool calling actually works, and the loop around it in the agent loop.

Which leaves the paper’s central question still live, and worth keeping in mind when your agent calls a tool it did not need: current models decide whether to call a tool from the schema description and their post-training, not from a measured estimate of whether the answer would help them. Toolformer’s criterion — does knowing this reduce my uncertainty — remains a better formulation of the decision than anything in a typical tool description, and nothing at the application layer currently computes it.