Function Calling With Llama 3: The JSON Convention Meta Documents

Llama 3.1 can call tools, and there is no API field involved anywhere. The model was fine-tuned to write a specific JSON shape into its ordinary text output when the prompt asks it to, and everything else — the schema you send, the tool_calls array you get back — is software in front of the weights translating in both directions.

There is no tool_calls field

If you call a hosted Llama through an OpenAI-compatible endpoint, you send a tools array and receive a tool_calls array, and it looks exactly like every other provider. That is real and usable, but it is worth knowing what it is: the server serialises your schemas into the prompt text, the model writes JSON into its completion, and the server parses that JSON back out into the field you expected. There is no separate channel. Every one of those bytes went through the same token stream as the prose.

This has consequences you will meet eventually. The tool schemas cost input tokens on every single request, and a large schema set is a large fixed cost per turn. The model can emit malformed JSON, because nothing constrains it to be valid unless the server also applies grammar-constrained decoding. And two servers can implement the parsing differently, so the same weights can appear to have different tool-calling reliability depending on who is hosting them.

Declaring tools in the system block

Meta documents two routes. Built-in tools — brave_search, wolfram_alpha, code_interpreter — are switched on with Environment: ipython and a Tools: line in the system header, and are covered in the built-in tools page. For your own functions the documented route is zero-shot: put the JSON schemas in the system or user message with an instruction about the reply format.

The convention Meta publishes for a custom tool looks like this inside the system block:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Environment: ipython
Cutting Knowledge Date: December 2023
Today Date: 11 August 2026

You have access to the following functions. To call a function,
respond with a JSON object in the format
{"name": function name, "parameters": dictionary of argument name
and its value}. Do not use variables.

{
  "type": "function",
  "function": {
    "name": "get_current_weather",
    "description": "Get the current weather for a location",
    "parameters": {
      "type": "object",
      "properties": {
        "location": {"type": "string",
                     "description": "City, e.g. Lima, Peru"},
        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
      },
      "required": ["location"]
    }
  }
}<|eot_id|><|start_header_id|>user<|end_header_id|>

What is the weather in Lima?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

“Do not use variables” is not filler. Without it the model has a documented tendency to emit {"location": location} — a plausible-looking function call with an unbound name in it — because that is what function calls look like in the code it was trained on. The reference prompt includes that instruction and you should keep it.

The prompt format for all of this is published at llama.com’s Llama 3.1 prompt format documentation.

What the model emits

For a custom tool, the assistant turn is a bare JSON object with exactly two keys:

{"name": "get_current_weather", "parameters": {"location": "Lima, Peru", "unit": "celsius"}}

Two details are easy to get wrong when you have come from another provider. The key is parameters, not arguments. And the value is a JSON object, not a JSON-encoded string — OpenAI’s format nests a string containing JSON, and Llama’s reference format does not. Code that runs json.loads on the value will fail here, and code that forgets to run it will fail there.

For a built-in tool the shape is different again: the model emits <|python_tag|> followed by a Python-like call expression such as brave_search.call(query="..."). If you are parsing generically, the <|python_tag|> token is your signal to switch parsers.

Why the call ends with eom_id

When Environment: ipython is set, a tool call terminates with <|eom_id|> rather than <|eot_id|>. The two tokens encode a real distinction: end of turn means the model has finished and is handing control back to the user; end of message means the model has finished this message but expects the conversation to continue without a user, because it is waiting for a tool result.

Get this wrong and the failure is specific and confusing. If your stop criteria include only <|eot_id|>, generation runs past the tool call and the model starts inventing the tool’s response — hallucinated weather data, formatted convincingly. If your stop criteria include only <|eom_id|>, ordinary conversational turns never terminate. Both tokens are stop tokens, and both must be registered. This is the same class of mistake as confusing eot_id with end_of_text, with a more expensive symptom.

Without Environment: ipython in the system block, the model uses <|eot_id|> for tool calls too. So whether <|eom_id|> appears at all depends on a line of text in your prompt, which is a good example of how little of this is enforced by anything.

Reliability is a function of model size

Because tool calling here is a formatting behaviour learned during instruct tuning rather than a constrained output mode, how well it works scales with the model in a way that a real API field would not. Meta documents tool use as a capability of the Llama 3.1 instruct models generally, and in practice the experience across the sizes is not the same thing at different speeds.

The failure modes shift as the model gets smaller. A large model tends to fail by choosing the wrong tool or filling a plausible-but-wrong argument — a reasoning failure, and one you can address with better descriptions in the schema. A small model tends to fail by producing something that is not the format at all: prose describing what it would call, a call wrapped in a markdown fence, JSON with the arguments inlined as a string, or the variable-name pattern the reference prompt explicitly warns against. Those are formatting failures, and the fixes are different.

Three things follow. If you must use a 1B or 3B model with tools, use grammar-constrained decoding rather than better prompting — it converts the entire formatting-failure class into a non-event by making malformed output unrepresentable. Keep the tool count low; a long list of schemas is both a large token cost and a harder selection problem, and splitting into a routing step followed by a small tool set often works better than one flat list. And do not assume a tool-calling integration that works against 70B will work against 8B without revisiting the prompt, because the layer that makes it work is the prompt.

Parsing it without getting burned

• Validate against the schema you sent, every time. Nothing guarantees the emitted object matches. Required keys go missing, enums get values outside the list, and types arrive as strings. Treat the output as untrusted input from a model that is trying to be helpful, not as a structured response.
• Handle prose around the JSON. The model may preface a call with a sentence of explanation. A parser that assumes the whole message is JSON will fail on a response that is otherwise correct.
• Do not assume one call per turn. Multiple calls may arrive in one message, and how they are delimited is not tightly specified — some deployments emit a JSON list, some emit consecutive objects, some emit one and stop.
• Send the result back in the right role. Tool output goes into an ipython-role message, not a user one; the details are in the tool result message format.
• Consider constrained decoding instead. If your runtime supports a JSON grammar or a schema constraint, applying it during generation removes the malformed-output class of failure entirely rather than catching it afterwards. This is the single largest reliability gain available here, and it is a runtime feature rather than a model one.