Function Calling vs Structured Output: Pick the Right One

Both send a JSON Schema. Both come back as JSON validated against it. Both are implemented with the same mask over the sampler. The difference is not in the mechanism at all — it is in who decides whether a call happens.

Underneath, they are the same thing

A tool definition is a name, a description and a JSON Schema for its arguments. A structured output request is a name, a description and a JSON Schema for the response. When the model produces a tool call, the arguments string is generated under the same constraint machinery that produces a response_format body. There is no deeper capability in one than the other.

You can therefore emulate either with the other. Force a single tool with tool_choice naming it and you have structured output on an API that predates the feature — this is the standard route on Anthropic models and the reason many codebases have a tool called record_answer that does nothing. Conversely, a response_format with an anyOf over several action shapes gets you something close to tool selection.

Both emulations work and both are slightly wrong, for the reason in the next section.

What actually differs

The last row is soft but real. Describing an extraction as a tool named save_invoice tells the model there is a world in which it does not save the invoice. If you always want the record, that ambiguity is pure downside — and it is exactly the ambiguity that produces the “model replied with prose instead of calling the tool” bug report.

The decision rule

One question, asked about your call site:

Is there any path through my code where the right outcome is
"the model does not produce this object"?

  no  -> structured output. One shape, always, no branch.
  yes -> tools. The model needs to be able to decline or to choose.

And a second, only if you answered "yes":

Does the model need the RESULT of the call to continue?

  yes -> tools, with the loop. This is the whole point of tools.
  no  -> tools with tool_choice: "required", or reconsider; you may
         have a router, not an agent.

Worked through the usual cases:

• Extraction, classification, scoring, reformatting. You always want the object. Structured output. This is the majority of production use and the majority of misuse.
• An assistant with a weather lookup and a calendar. The model must decide whether the question needs either, and it needs the answer to reply. Tools, with the loop.
• Routing to one of five handlers. A closed choice with no execution result needed. A structured output with an enum beats five tool definitions: it is smaller in context, it cannot return zero choices, and it gives you a probability distribution over the branches if you ask for logprobs.
• “Answer, or say you need to search.” Genuinely a model decision. Tools, or a structured output whose first field is an enum of answer / needs_search — both are defensible, and the structured version is easier to test.

Why everyone conflates them

Function calling arrived first, in June 2023, and for over a year it was the only way to get schema-enforced JSON from a hosted model. An entire generation of libraries — and a great deal of tutorial content still in the search results — is built on a forced single tool used purely as a JSON extractor. That code is not wrong, it was correct at the time, and its shape is why the two features feel like one.

Since dedicated structured-output modes shipped (OpenAI in August 2024, with comparable features following elsewhere), the fake-tool pattern costs you clarity for no benefit. If you find a tool named extract or output or respond_with_json in your codebase, it is a candidate for replacement — with the caveat that on some APIs the forced tool call remains the supported path, so check before you migrate.

Using both in one call

They are not mutually exclusive, and the combination has a sharp edge. A request can carry tools and a response schema, but the schema applies to the assistant’s message content, not to the tool arguments — so on the turns where the model calls a tool, there is no content for the schema to constrain, and on the turn where it finally answers, there is. Some stacks reject the combination outright rather than reasoning about it.

The clean version is to keep them in separate calls: run the tool loop until the model is done, then make one final call with no tools and a strict response schema to produce the record. It costs one extra call and removes an entire category of “why is the schema sometimes not applied” question. It also lets the final call use a smaller, cheaper model, since summarising a finished transcript into a fixed shape is a much easier task than the loop was.

The separation has a second benefit that shows up when things go wrong. One call doing both jobs gives you one failure — “the run produced a bad record” — with no way to tell whether the tools were used badly or the summary was written badly. Split, the two are separately observable and separately testable: the loop is judged on whether it gathered the right information, the final call on whether it shaped it correctly, and a regression in either is attributable without anyone reading a transcript.