Writing Tool Descriptions the Model Actually Understands

A tool description is the only thing standing between a model and the wrong function. It is read once per request, competes for attention with every other description, and is written by an engineer in the thirty seconds after the function works. That asymmetry is where most agent misbehaviour comes from.

The description is a prompt

Whatever your SDK’s ergonomics suggest, the schema is not metadata. It is serialised into the context window as text and read by the model with the same machinery that reads the system prompt. If you would not accept a sentence in your system prompt, do not accept it in a description.

Two consequences follow immediately. First, a description that is precise about when to use this is worth more than one that is precise about what the function does — the model is choosing among alternatives, not reading API docs. Second, descriptions have a cost: they are input tokens on every iteration of the loop, so verbosity is billed repeatedly. Neither observation supports the common practice of copying the function’s docstring in unchanged.

Six rules that survive contact

• Write the boundary, not just the behaviour. The sentence that earns its tokens is usually the one starting “Use this when…” or “Do not use this for…”, naming the sibling tool that covers the other case. Selection errors are relative, so the fix is relative.
• Constraints go in the schema, not the prose. A pattern, an enum, a minimum, an explicit format — these are machine-checkable and, where the provider uses constrained decoding, physically prevent the invalid value. “The date should be ISO 8601” in the description is a suggestion; “format”: “date” is not.
• Name parameters after domain concepts. customer_email beats arg2 and also beats email when there is a support agent’s address in scope. The model has no type checker; the name is the type.
• Say what the tool returns. A model that does not know a search returns ten snippets rather than a document will call it once and answer from nothing. One clause — “returns up to 10 results with title, url and a 200-character excerpt” — changes the plan.
• Encode the side effects. If it writes, sends, charges or deletes, the first clause of the description should say so. Models are noticeably more conservative with tools whose irreversibility is stated, and it costs one sentence.
• One tool, one job. A tool with a mode parameter that switches between three behaviours is three tools wearing a coat, and the model will pick the wrong mode.

Three rewrites

1. The docstring copy

BEFORE
  name: "query_db"
  description: "Runs a query against the database and returns rows."
  parameters: { sql: { type: "string" } }

AFTER
  name: "run_analytics_query"
  description:
    "Run one read-only SQL SELECT against the analytics warehouse
     (Postgres, schema 'events'). Returns at most 200 rows as JSON.
     Use for aggregate questions over event history. Do NOT use for
     current account state -- use get_account for that. Writes are
     rejected."
  parameters:
    sql:        { type: "string",
                  description: "A single SELECT statement. No semicolons,
                                no CTEs writing data, no DDL." }
    max_rows:   { type: "integer", minimum: 1, maximum: 200, default: 50 }

Four separate changes. The name says which database. The dialect and schema are stated, so the model stops writing MySQL syntax. The negative boundary points at the sibling tool that was previously losing half its traffic. And the row cap moved from an undocumented truncation into a parameter, so a model that needs more can ask — and the schema bounds what it can ask for.

2. The ambiguous pair

BEFORE
  search_docs   "Search the documentation."
  search_kb     "Search the knowledge base."

AFTER
  search_public_docs
    "Search the published product documentation that customers can
     read. Covers API reference, guides and changelog. Use for
     'how does X work' questions. Returns 10 excerpts with URLs you
     may cite to the customer."
  search_internal_runbooks
    "Search internal runbooks and incident postmortems. NOT customer
     visible -- never quote verbatim to a customer. Use for
     diagnosing a failure or finding an escalation path. Returns 10
     excerpts with an internal-only wiki link."

“Documentation” and “knowledge base” are synonyms to a model, and no amount of system-prompt instruction reliably fixes a tie at the point of selection. The rewrite makes the distinction be about audience, which is a concept the model can apply to a user’s question. The confidentiality clause is a bonus: it is the only place a customer-facing model will reliably read it.

3. The dangerous one

BEFORE
  name: "update_subscription"
  description: "Update a customer's subscription."
  parameters: { customer_id, plan, prorate }

AFTER
  name: "change_subscription_plan"
  description:
    "IRREVERSIBLE BILLING CHANGE. Moves a customer to a different plan
     immediately and issues a proration charge or credit on the spot.
     Cannot be undone by calling this again -- a downgrade after an
     upgrade produces two invoice lines, not a rollback. Requires that
     you have already called get_subscription in this conversation and
     confirmed the current plan. If the customer has not explicitly
     named the target plan, ask them instead of calling this."
  parameters:
    customer_id:      { type: "string", pattern: "^cus_[A-Za-z0-9]+$" }
    target_plan:      { type: "string", enum: ["basic","pro","scale"] }
    confirmed_by_user:{ type: "boolean",
                        description: "True only if the user named this
                                      exact plan in their own words." }

The enum makes an invented plan name impossible rather than unlikely. The read-before-write precondition is stated as a precondition. And confirmed_by_user is a deliberate trick: it does not enforce anything by itself — the model can lie — but it makes the confirmation an explicit, loggable claim your handler can gate on, which is the hook an approval gate hangs from.

Measuring your own selection accuracy

Every rewrite above is an argument, not a result. Whether it helps your tool set against your traffic is an empirical question with a cheap answer, and published numbers for somebody else’s tools cannot substitute for it. The harness is about forty lines and one afternoon of labelling:

# cases.jsonl -- 60+ real user turns, labelled by hand.
# {"turn": "where's my order 90210", "expect": "get_order_status"}
# {"turn": "how do refunds work",     "expect": "search_public_docs"}
# {"turn": "hi",                      "expect": null}   # <- no tool

import json, collections

def selected(turn, tools):
    msg, _ = call_model([{"role": "system", "content": SYSTEM},
                         {"role": "user", "content": turn}],
                        tools=tools, temperature=0)
    calls = msg.get("tool_calls") or []
    return calls[0]["function"]["name"] if calls else None

def score(cases, tools, runs=3):
    confusion = collections.Counter()
    for c in cases:
        for _ in range(runs):                 # temp 0 is not deterministic
            got = selected(c["turn"], tools)
            confusion[(c["expect"], got)] += 1
    total   = sum(confusion.values())
    correct = sum(n for (e, g), n in confusion.items() if e == g)
    return correct / total, confusion

before, cm_b = score(cases, TOOLS_V1)
after,  cm_a = score(cases, TOOLS_V2)
print(round(before, 3), "->", round(after, 3))

Three details that matter more than the code. Include expect: null cases — turns where no tool should fire — or you will optimise your way into a system that always calls something. Repeat each case, because temperature 0 is near-deterministic and not deterministic, and a single run turns noise into a conclusion. And label from real traffic; invented test turns are written in the vocabulary of the tool descriptions, which is precisely the bias you are trying to measure away.

Reading the confusion matrix

The aggregate accuracy is the least interesting output. What you want is the off-diagonal:

• A dense cell between two tools — they are not distinguishable. Rewrite them as a pair, adding a mutual boundary clause to each. Fixing one alone tends to move the error to the other direction.
• A row that scatters everywhere — that tool’s trigger condition is not expressible from the user’s words. Often the real fix is that it should not be a top-level tool at all.
• Errors concentrated in the null column — the model is answering from its own knowledge instead of calling. That is a system-prompt problem, not a description problem.
• Errors concentrated in the null row — it calls tools when it should chat. Check for tool_choice: “required” left on by accident.

Run the harness once before touching anything, keep the case file in the repo next to the tool definitions, and re-run it whenever you add a tool. Adding a tool changes the accuracy of every existing one, which is the subject of how many tools is too many.