Function Calling Support in Phi: Which Versions Added It

Most of the Phi family cannot call functions, in the sense that most people mean by the phrase. One line can. The difference is visible in the tokenizer, which makes it easy to check and impossible to fake.

Which releases document it

release                    native tool tokens   documented tool calling
Phi-2                      no                   no
Phi-3-mini / small / medium no                  no
Phi-3.5-mini               no                   no
Phi-3.5-MoE                no                   no
Phi-3.5-vision             no                   no
Phi-4 (14B)                no                   no
Phi-4-mini-instruct        yes                  yes
Phi-4-multimodal-instruct  yes                  yes

The Phi-4-mini line is where Microsoft added tool calling to the family as a documented capability, with dedicated tokens and a specified request format on the model card. Everything earlier can be coaxed into emitting JSON that you then parse, which is a different thing and fails differently.

You can verify the row for any checkpoint yourself in three lines, which is more reliable than any table including this one:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("microsoft/Phi-4-mini-instruct")
print([t for t in tok.get_added_vocab() if "tool" in t.lower()])

What “native” means here

Tool calling is not a capability bolted onto a model. It is two things together: reserved vocabulary entries that mark a span as a tool call rather than as prose, and post-training that taught the model to emit those markers when a tool is appropriate and not otherwise.

The markers are what make the channel unambiguous. When a model with native support emits its tool-call open token, a server can switch from streaming text to buffering a call with no heuristics involved. When a model without them emits {"name": "get_weather", your parser has to guess whether that is a call, an example inside an explanation, or a fragment of a code block. On a 3B model that ambiguity is not theoretical — small models are markedly more likely to narrate what they are about to do before doing it.

The training half matters just as much. A model that has seen tool schemas during post-training learns when not to call: it answers “what is 2+2” directly instead of reaching for a calculator tool that happens to be in scope. A prompted model with no such training tends to either call constantly or never, and which one you get depends on wording.

There is a third element that is easy to miss, which is the return path. A tool result has to re-enter the conversation as something other than a user turn, or the model treats the JSON you fed back as a message from the person it is talking to. Native support gives that its own role; a prompt-based scheme has to fake it, usually by labelling the result inside a user turn — a compromise the model was never trained on, and a common reason a small model calls the same tool again immediately after receiving its answer.

The Phi-4-mini tool format

Microsoft’s Phi-4-mini-instruct card documents tool definitions delivered in the system turn between <|tool|> and <|/tool|>, calls emitted by the assistant between <|tool_call|> and <|/tool_call|>, and results fed back in a <|tool_response|> turn. Rendered, a single-tool conversation looks like this:

<|system|>
You are a helpful assistant with access to tools.
<|tool|>[{"name":"get_weather","description":"Current weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}]<|/tool|><|end|>
<|user|>
What is the weather in Amsterdam?<|end|>
<|assistant|>
<|tool_call|>[{"name":"get_weather","arguments":{"city":"Amsterdam"}}]<|/tool_call|><|end|>
<|tool_response|>
[{"temperature_c":14,"conditions":"rain"}]<|end|>
<|assistant|>

Note that the tool list is JSON inside the system turn rather than a separate API field, and that the tool response comes back as its own role. As always, render this with tokenizer.apply_chat_template and pass your tools through the tools= argument rather than concatenating strings — see Phi’s chat template and special tokens for why hand-rolling this is worse than it looks.

The workaround for the rest

For Phi-3, Phi-3.5 and the 14B Phi-4, you are writing the protocol yourself. The version that survives contact with a small model has four properties:

• One schema, in the system turn. Not a list of ten tools. Selection among many options is the part small models are worst at; do the selection in your own code and present the model with the one candidate.
• A delimiter you can find. Ask for the call inside a fenced block with a distinctive tag, so extraction is a regex rather than a JSON scan of the whole reply.
• A documented no-call path. Give the model an explicit way to decline — a literal token to emit when no tool applies. Without one, “always output a call” is the only instruction it has.
• A prefill. End the prompt part-way into the expected output so the first token is already inside the structure.

Constrained decoding is the real fix

Prompting for JSON asks the model to be well-formed. Constrained decoding makes malformed output unrepresentable: at each step the sampler masks out every token that could not continue a valid document under your schema. Since you are running these weights yourself, this is available to you in a way it is not on most hosted APIs.

# vLLM, OpenAI-compatible server
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "microsoft/Phi-3-mini-4k-instruct",
  "messages": [
    {"role": "system", "content": "Emit a call to get_weather, or {\"tool\": null}."},
    {"role": "user", "content": "Weather in Amsterdam?"}
  ],
  "max_tokens": 128,
  "temperature": 0,
  "guided_json": {
    "type": "object",
    "properties": {
      "tool": {"type": ["string", "null"], "enum": ["get_weather", null]},
      "arguments": {"type": "object", "properties": {"city": {"type": "string"}}}
    },
    "required": ["tool"]
  }
}'

The same idea appears as GBNF grammars in llama.cpp and as regex or JSON-schema guides in Outlines. It fixes syntax completely and semantics not at all: a schema-valid call to the wrong tool with invented arguments is still schema-valid. Validate the arguments against reality before you execute anything.

Deciding whether it is good enough

Whether a small model can do your tool calling is not answerable in general and is cheap to answer specifically. The four failure modes worth counting separately, because they have different fixes, are:

• Malformed output — not parseable at all. Fixed completely by constrained decoding, so if this dominates your failures you have a configuration problem rather than a model problem.
• Wrong tool — a valid call to the wrong function. Fixed by narrowing the candidate set in your own code before the model sees it, or by splitting one over-broad tool into two clearly distinct ones.
• Hallucinated arguments — a plausible city name that was never mentioned, an id invented to fill a required field. Fixed by validation against the real world, and by making optional fields genuinely optional in the schema so the model is not forced to produce one.
• Should not have called — reaching for a tool on a question it could answer directly, or the reverse. This is the one that does not respond to prompting on a small model, and the one that most often decides you need the tuned variant.

Thirty to fifty real requests from your own traffic, labelled by hand with the call you wanted, is enough to see which of the four dominates. That is an afternoon of work, and it replaces an argument about model choice with a number.