Function Calling in Qwen: The Hermes-Style Tool Call Convention

Qwen has no tool-calling API at the weights level. What it has is a convention, borrowed from the Nous Research Hermes models, in which tool definitions are injected into the system prompt as JSON inside XML tags and the model is trained to reply in the same shape. Everything your SDK shows you as a structured tool_calls array is a parser on top of that text.

Tool calling is a prompt convention

This is worth internalising before looking at the format, because it explains every one of the failure modes. A model that “supports function calling” is a model that was post-trained on examples in a particular textual shape, and that will therefore reproduce that shape with high reliability when prompted in it. There is no separate output channel and no schema enforcement in the decoder unless your serving stack adds constrained decoding on top. The model can emit malformed JSON, invent a function that was not offered, or wrap a call in prose, because all of those are sequences its sampler can produce.

Qwen adopted the Hermes convention rather than inventing one, which is why vLLM’s tool parser for Qwen is literally called hermes. The convention originates with Nous Research’s Hermes 2 Pro model card, and Qwen’s implementation of it lives in the Jinja template in tokenizer_config.json of every Qwen2.5 and Qwen3 instruct repository — which means you can read the authoritative version of what follows by opening that file.

What the template injects

When you pass a tools argument to apply_chat_template, the template appends a block to the system turn. For a single weather function, the rendered system turn comes out as:

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name, e.g. Lisbon"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["city"]}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>

Several things in that block are load-bearing and easy to break if you are constructing the prompt yourself:

• The tool schemas are OpenAI-shaped — {"type": "function", "function": {...}} with a JSON Schema under parameters. You do not need to convert anything if your definitions already target the OpenAI API.
• Each schema is one line of JSON, newline-separated inside the <tools> tags. Multiple tools mean multiple lines, not a JSON array.
• The instruction text is itself part of what the model was trained on. Paraphrasing it — writing your own “you may call functions” preamble — moves you off the trained distribution and reliability drops. Use the template.
• The block goes in the system turn. If you had no system message, the template creates one; if you had one, the tools are appended to it.

Every one of those tool schemas is prompt tokens, charged on every request in the conversation. A dozen tools with thorough descriptions is comfortably a thousand tokens of overhead per turn, which is the usual reason a tool-using agent costs several times what its conversation length suggests.

What the model emits

Given the block above and the user message “What’s the weather in Lisbon?”, the raw completion is text, and looks like this:

<tool_call>
{"name": "get_weather", "arguments": {"city": "Lisbon", "unit": "celsius"}}
</tool_call><|im_end|>

Two calls in one turn are two consecutive <tool_call> blocks, separated by a newline, inside the same assistant turn — see the page on Qwen’s parallel tool calls for what that means for your dispatch loop. And note that a model can legitimately emit prose and a tool call in the same turn: the convention does not forbid text before the first <tool_call> tag, so a parser that assumes the whole completion is either prose or a call will lose content.

The arguments value is a JSON object here, not a JSON-encoded string. This differs from OpenAI’s wire format, where arguments is a string you have to parse a second time. Any adapter converting between the two has to serialise or deserialise at that boundary, and forgetting to is the source of the classic double-encoded "{\"city\": \"Lisbon\"}" bug.

Feeding the result back

This is the part that surprises people, because the obvious guess is wrong. There is no tool role in the rendered prompt. The template folds a tool result into a user turn wrapped in <tool_response> tags:

<|im_start|>assistant
<tool_call>
{"name": "get_weather", "arguments": {"city": "Lisbon", "unit": "celsius"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
{"city": "Lisbon", "temp_c": 19, "conditions": "clear"}
</tool_response><|im_end|>
<|im_start|>assistant

At the message-list level you still write {"role": "tool", "content": "..."} and let the template do the folding — the point of knowing this is diagnostic. If you are debugging why a model ignored a tool result, render the prompt and look at it: a result that ended up in its own <|im_start|>tool turn means something in your stack is not using Qwen’s template, and the model is reading a role name it has never been trained on. The turn structure itself is covered in the ChatML template page.

When several tool results come back at once, they are folded into a single user turn with one <tool_response> block each, in the order the calls were made. Preserving that order matters: the model is matching results to calls positionally, since the convention carries no call id.

Turning it into an OpenAI-shaped response

If you serve Qwen yourself and want tool_calls in the response body rather than tags in the content, the serving framework has to be told which parser to use. In vLLM that is two flags:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

Without them the server is not wrong, it is just literal: the tool definitions still reach the model, the model still emits the tags, and they arrive in choices[0].message.content as text while tool_calls stays null. That is the exact symptom to look for when an SDK reports no tool calls from a model that is visibly trying to make one.

Streaming is where these parsers earn their keep and where they are most fragile. The parser has to recognise a partial <tool_call> tag across chunk boundaries and decide whether to emit the text as content or buffer it as an incipient call — so a model that stalls mid-tag can produce a stream that emits nothing for several hundred milliseconds and then a complete call at once. Alibaba documents its hosted behaviour separately in the Alibaba Cloud Model Studio documentation; the hosted OpenAI-compatible endpoint does this parsing for you.