Tool Calling: How the Model Picks a Function

“The model calls a function” is a useful lie that becomes a liability the first time a call is malformed. The model emits tokens. Everything else — the parsing, the dispatch, the object your SDK hands you — is written by somebody, and knowing by whom is how you fix it.

The model does not call anything

A model is a function from a token sequence to a distribution over the next token. It has no network access and no runtime. When you “give it tools”, three things happen, none of which involve execution:

• Your tool definitions are serialised into the model’s context as text, in whatever format that model was trained on. They are input tokens, and you pay for them on every request in the loop.
• The model emits a structured span — special tokens, a JSON object, or both, depending on the family — indicating a tool name and arguments.
• The provider parses that span out of the raw output and returns it in a normalised field, so you never see the model-specific syntax.

That third step is why the same code works across providers and why the failure modes are so confusing: the normalisation is doing real work, and when it fails you get an empty tool_calls array and a content field full of JSON the model clearly intended as a call.

What actually goes over the wire

The request. Note that tools sits beside messages as a peer, not inside the conversation — the provider is responsible for rendering it into the prompt:

{
  "model": "some-capable-model",
  "messages": [
    {"role": "system", "content": "You answer questions about orders."},
    {"role": "user", "content": "Where is order 90210?"}
  ],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_order_status",
      "description": "Current fulfilment status and carrier tracking for one
                      order. Use only when the user supplies an order id; use
                      search_orders when they describe the order instead.",
      "parameters": {
        "type": "object",
        "properties": {
          "order_id": {"type": "string", "pattern": "^[0-9]{5,12}$"},
          "include_history": {"type": "boolean", "default": false}
        },
        "required": ["order_id"],
        "additionalProperties": false
      }
    }
  }],
  "tool_choice": "auto"
}

Everything inside function is prompt material. The name, the description, every property description, the enum values — all of it is text the model reads, and all of it is billed as input tokens. A schema’s description fields are not documentation for your team; they are the instructions. This is the whole reason tool description design is a discipline rather than a style preference.

tool_choice is the one field that changes the mechanism rather than the content. “auto” lets the model answer in prose or call. “required” forces a call — useful for extraction, dangerous in a loop, because a model with nothing left to do will call something anyway. Naming a specific function forces that one, which turns the model into a parameter extractor and is often the right call for a step you already know you need.

How the JSON stays well-formed

Two mechanisms, and providers differ in which they use. The weak one is training: models are fine-tuned heavily on tool-call formats and emit them correctly most of the time. The strong one is constrained decoding — at each step the sampler masks out every token that could not continue a valid parse of the schema, so an invalid character is assigned zero probability rather than a low one.

The distinction is observable. Under constrained decoding, structural errors go to zero — you will never see an unclosed brace — while semantic errors persist untouched: a required field filled with a plausible invention, an enum value chosen because it sounded right, a date in the wrong timezone. Grammar constraints guarantee shape and say nothing about truth. If your validation only checks that the JSON parses, constrained decoding has made your validation useless without making your data correct.

One practical consequence: a tight schema is not just documentation, it is a decoding constraint. A “enum” of four strings removes every other string from the sample space. A free-text field removes nothing.

What comes back, field by field

{
  "choices": [{
    "finish_reason": "tool_calls",
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_8bR2xk",
        "type": "function",
        "function": {
          "name": "get_order_status",
          "arguments": "{\"order_id\":\"90210\"}"
        }
      }]
    }
  }],
  "usage": {"prompt_tokens": 412, "completion_tokens": 23}
}

• finish_reason: “tool_calls” is the branch condition. Check this rather than truthiness of the array — some providers include an empty array.
• id is generated by the provider, not the model, and exists only to pair the result back. You must echo it exactly in the tool_call_id of your reply.
• arguments is a string containing JSON, not an object. This trips up everyone once. It is a string because it is generated text and may not parse — the API cannot promise you an object it might not be able to build.
• content is often null, but not always. Some models emit reasoning alongside a call. Do not assume one excludes the other, and do not drop the content when you append the message back.
• tool_calls is an array because a model may request several at once, which is a genuine behaviour change and not just a container.

Streaming makes it harder

Under stream: true the call arrives in fragments. Each chunk carries a delta with an index, and the arguments string arrives a few characters at a time across many chunks. You must accumulate per index, and the accumulated string is not parseable until the stream for that index completes. The name usually arrives in the first fragment and is then omitted from the rest.

The bug this produces is subtle: code that calls json.loads on each delta appears to work in testing because short arguments arrive in one chunk, then fails in production the first time a model writes a long string argument. If you are streaming an agent loop, buffer tool calls to completion and stream only the assistant’s prose.

Where it goes wrong

The pattern across all five: the boundary between what the model produced and what your stack produced is where the bug lives. Log the raw response body, not the SDK object, for exactly this reason — the SDK object has already normalised away the evidence.