How to Count Tokens Before You Send a Request

Counting before you send turns a class of production 400s into a local branch you control. It is fifteen minutes of work and it is the difference between truncating deliberately and having the API truncate for you.

Why count at all

Four reasons, in descending order of how much trouble they save. You need to know whether the request will fit, so you can trim rather than fail. You need to compute a safe max_tokens, which depends on the prompt length. You need per-feature cost attribution before the money is spent rather than after. And you need to enforce user-facing limits in the unit those limits are actually denominated in — a character cap is a different budget in every language.

The distinction that organises the rest of this page is between estimation and admission control. Estimation answers “roughly how much will this cost” and may be wrong by 20% without anything bad happening. Admission control answers “will this request be accepted” and must not be wrong at all, because being wrong means a 400 in front of a user. Different accuracy requirements justify different techniques, and conflating them is why so much code either calls a network endpoint on every request or divides the character count by four and hopes.

Three ways to count

Tokenizer libraries, for the OpenAI families

tiktoken is the reference implementation, and the important part is choosing the encoding rather than guessing it:

import tiktoken

enc = tiktoken.encoding_for_model("<model name>")   # resolves the encoding
# falls back explicitly if the model is unknown to your tiktoken version:
# enc = tiktoken.get_encoding("o200k_base")

n = len(enc.encode(text))

Note the failure this avoids: encoding_for_model raises KeyError for models newer than your installed tiktoken, which is a common and confusing break after a model launch. Pin the fallback deliberately instead of letting the exception reach production.

Hugging Face tokenizers, for open-weights families

For anything with a published tokenizer, load the real one — and critically, count the templated string, not the raw messages:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("<repo id>")

# WRONG: counts the content and none of the scaffolding
n_wrong = sum(len(tok.encode(m["content"])) for m in messages)

# RIGHT: counts exactly what the server will receive
n = len(tok.apply_chat_template(messages, add_generation_prompt=True))

The gap between those two numbers is the per-message template scaffolding, and on a long conversation of short turns it is not small — every turn carries role markers and end-of-turn tokens.

Provider count endpoints, for closed models

Where the tokenizer is not published, the provider exposes a counting endpoint that runs the real one: Anthropic’s /v1/messages/count_tokens and Google’s countTokens both take the same request body you would send and return the input count without running inference. They are the only exact option for those families, and they cost a network round trip, so the sensible pattern is to use them to calibrate a local estimator rather than to call them on every request.

Note the asymmetry in what they can tell you: a count endpoint counts input. Nothing can count output in advance, because the output does not exist yet. Any budget that depends on knowing the completion length is a budget built on a cap you enforce, not on a number you measured.

Outside Python

The tokenizer ecosystem is Python-first and the alternatives are worth knowing before you discover them at build time. In JavaScript and TypeScript, js-tiktoken is a pure-JS port suitable for edge and browser use, while the WASM bindings are faster and considerably heavier; @huggingface/transformers can load a repository’stokenizer.json and apply its chat template the same way the Python library does. There are maintained ports for Go, Rust and the JVM as well.

Two operational details bite people in every one of these environments. Tokenizer construction is expensive — it parses a large merge table — so build it once at process start and reuse it, never per request. And the vocabulary files are large enough to matter in a serverless bundle or a browser payload, which is the usual reason a team ends up estimating client-side and counting exactly on the server. That split is fine, provided the client’s estimate is deliberately conservative rather than accidentally optimistic.

What a naive count misses

An admission gate

Put the estimate and the exact count in different places. Estimate to decide how much to retrieve; count exactly to decide whether to send:

SAFETY = 128   # template scaffolding + provider-side additions

def estimate(text):
    "Cheap pre-filter only. Never the admission check."
    return len(text.encode("utf-8")) // 4

def admit(messages, tools, window, output_cap, want_out):
    used = count_exact(messages, tools)          # one of the three above
    room = window - used - SAFETY
    if room < want_out:
        trimmed = trim_to(messages, window - SAFETY - want_out)
        used = count_exact(trimmed, tools)
        room = window - used - SAFETY
        if room < 1:
            raise PromptTooLong(used, window)
        messages = trimmed
    return messages, min(want_out, output_cap, room)

The trimming step is where the interesting decisions live, and they are the subject of truncation strategiesNote that the gate re-counts after trimming rather than assuming the trim hit its target. Trimming removes whole messages, so the result overshoots by an unpredictable amount, and a second count is far cheaper than a rejected request. Note too that it returns the messages it actually intends to send: a gate that validates one list and a caller that sends a different one is a bug that only appears under the conditions the gate was written for. The residual gap between your count and the provider’s is covered in why the numbers differ.