Tool Schemas That Cost You Tokens

Your tool definitions are in the prompt on every turn of the loop, they do not appear in your source as text, and no tokeniser you run locally will tell you how many tokens they became. The only reliable measurement is a subtraction the API does for you.

The tokens you cannot see

Tool definitions do not reach the model as the JSON you sent. Providers serialise them into their own prompt format — different wrappers, different key ordering, sometimes a natural-language rendering — and then count that. Running tiktoken over your schema gives you a number that is in the right order of magnitude and is not the number you are billed for.

Three properties make this worth attention rather than a shrug. It is paid on every request, including every turn of an agent loop, so a twelve-step run pays it twelve times. It grows as your product does, one tool at a time, and nobody reviews the total. And it is charged at input price on tokens that are identical across all your traffic, which makes it the most cacheable spend you have.

Measuring it exactly

Two identical requests, one with tools and one without. The difference in usage.prompt_tokens is the true cost of your tool block on that provider, that day:

import os, json
from openai import OpenAI

client = OpenAI(base_url=os.environ.get("BASE_URL"), api_key=os.environ["API_KEY"])
MODEL = os.environ["MODEL"]
MSGS = [{"role": "user", "content": "hi"}]        # deliberately tiny

def prompt_tokens(**kw) -> int:
    r = client.chat.completions.create(model=MODEL, messages=MSGS, max_tokens=1, **kw)
    return r.usage.prompt_tokens

def cost_of(tools: list) -> int:
    return prompt_tokens(tools=tools) - prompt_tokens()

TOOLS = json.load(open("tools.json"))

print("all tools:", cost_of(TOOLS), "tokens")
for t in TOOLS:                                    # per-tool, one at a time
    name = t["function"]["name"]
    print(f"  {name:<28} {cost_of([t]):>5}")

# Ablations worth running on your largest tool:
stripped = json.loads(json.dumps(TOOLS))
for t in stripped:
    for p in t["function"].get("parameters", {}).get("properties", {}).values():
        p.pop("description", None)
print("without parameter descriptions:", cost_of(stripped))

Run it against every provider you route to. The same tool block measured on two providers will not give the same number, and if one of them is your failover route, its number is the one that applies during an incident — when your traffic is also unusual.

Two cautions on method. The per-tool sum will not equal the all-tools figure, because the block has a fixed wrapper; the difference is that overhead and it is worth knowing separately. And keep the message tiny and identical across both calls, or you are measuring your prompt.

Turning tokens into money

Substitute your own numbers; the prices below are illustrative. Say the diff reports 1,850 tokens for the tool block, input is priced at $1.00 per million tokens, and the service makes 200,000 model calls a month with an average of 2.4 turns per conversation:

turns/month  = 200,000 * 2.4                    = 480,000
tool tokens  = 480,000 * 1,850                  = 888,000,000
cost         = 888,000,000 / 1e6 * $1.00        = $888 / month

after trimming the block to 900 tokens:
             = 480,000 * 900 / 1e6 * $1.00      = $432 / month     (-$456)

with a prompt cache covering the tool block at 10% of input price,
assuming 85% of turns hit it:

  uncached   = 480,000 * 0.15 * 900 / 1e6 * 1.00   = $64.80
  cached     = 480,000 * 0.85 * 900 / 1e6 * 0.10   = $36.72
  total                                            = $101.52 / month

The shape of that result is the point, and it survives whatever your real prices are: caching beats trimming, and doing both beats either. It also tells you where the effort goes — an afternoon spent making the tool block cacheable, by keeping it byte-identical and at the front of the prompt, is worth more than a week of shortening descriptions.

The trims, in order of effect

• Send fewer tools. By a wide margin the largest win, and the one that also improves accuracy. Most applications have a state machine hiding in them: a user who has not authenticated cannot use nine of your fourteen tools. Filter the array per request. A model choosing among four tools is also more accurate than one choosing among fourteen, so this is free twice.
• Make the block cacheable. Identical bytes, stable ordering, at the front. A tool array built from a dict in a language without guaranteed iteration order will silently reorder and miss the cache; sort it.
• Collapse near-duplicate tools. Four tools that differ by one enum value are one tool with a parameter, at roughly a quarter of the tokens and with fewer ways for the model to pick wrong.
• Prune parameters you do not use. Tool schemas accumulate optional parameters nobody has passed in a year. Grep your handler for each one before deleting, then delete.
• Shorten descriptions last, and carefully. This is where people start, and it is the trim with the worst ratio of tokens saved to risk taken. Remove examples and restatements; keep the sentence that distinguishes this tool from its neighbour.

What not to trim

Names. get_customer_orders costs perhaps three more tokens than gco and is the single strongest signal the model has about what the tool does. The same holds for parameter names — start_date over d1, every time.

The disambiguating sentence. If two tools are ever confused for each other, the line in each description that says which is which is doing more work than everything else in the block. Trimming it saves twenty tokens and buys you a wrong tool call, which costs a full extra turn — about two thousand tokens on the arithmetic above.

And the enum values that close a parameter. It is tempting to replace a fifteen-member enum with a free string and a sentence describing the options, and it does save tokens. It also gives up the constraint, which means normalising the model’s answer forever. Enums are one of the few places in the block where the tokens buy you a guarantee rather than a nudge.

A last piece of accounting hygiene: measure the block again after every release that adds a tool, and put the number in the same dashboard as your token spend. This is a cost that grows by accretion — nobody ever decides to double it, and it doubles anyway over a year of one-tool pull requests. A single number, tracked, is enough to make the conversation happen before the invoice does.