Tokenizer Mismatch: The Bug That Only Appears in Production

A tokenizer mismatch is when the count your code computes and the count the provider bills disagree. It rarely raises an exception. It shows up as a context overflow on a request you calculated as safe, a cost forecast that is 20% under, or a truncation that removes the end of the document you most needed — all of which appear in production and not in your tests, because your tests use short inputs.

What the bug looks like

• A 400 for context length on a request you sized. You computed 100,000 against a 128,000 window and the provider reports 131,000. The gap is not random.
• A bill that exceeds the forecast by a consistent ratio. Consistency is the diagnostic: a systematic multiplier is a counting bug, whereas volume surprises are lumpy.
• Truncation that removes the wrong thing. Code that trims to a token budget with the wrong tokeniser trims too little — and then the provider truncates instead, from whichever end it chooses, which is usually not the end you would have chosen.
• Failures concentrated in one language. If the errors cluster in Japanese, Arabic or Thai content, the estimator is probably a characters-divided-by-four heuristic, which is approximately right for English and badly wrong elsewhere.

Six places the two counts diverge

1. The wrong tokeniser family. Model families use different vocabularies, and vocabularies are not interchangeable even when the token counts are close on average. Counting one family’s text with another family’s vocabulary gives an answer that is right on English prose and wrong on code, identifiers, numbers and non-Latin scripts — which is exactly the content your long requests contain.
2. The wrong version within a family. Vocabularies are revised between model generations. A tokeniser library pinned two years ago counts a current model incorrectly, and nothing warns you.
3. Counting the strings instead of the rendered prompt. The single largest contributor. See below.
4. Tool schemas not counted at all. They are serialised into the prompt and billed, and almost no estimator includes them. Twenty tools is routinely thousands of tokens.
5. Images counted as text, or not counted. Image input is billed by a tile-based formula that has nothing to do with character counts. If your requests contain images, no text tokeniser will ever reconcile.
6. An alias resolving to a different model. You count for the model you think you are calling. If the name is an alias and it moved, you are counting for the wrong one — see silent model updates.

The chat template is most of the gap

A model does not receive a list of message objects. It receives one flat token sequence, produced by a chat template that inserts special tokens marking each turn’s start, its role, and its end. Those tokens are counted and billed like any other.

# Roughly what a chat template produces (delimiters vary by family)
<|start|>system<|sep|>You are helpful.<|end|>
<|start|>user<|sep|>Hello<|end|>
<|start|>assistant<|sep|>

Counting only "You are helpful." and "Hello" misses every delimiter.
The overhead is a small fixed number of tokens PER MESSAGE — which is
negligible for one message and substantial for a 200-turn conversation.

With open weights you can render the template exactly and count the result, which removes the guesswork entirely:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(MODEL_ID)

rendered = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,   # include the assistant turn opener
)
n = len(tok(rendered, add_special_tokens=False).input_ids)

Two flags there are load-bearing and both are common sources of an off-by-a-few-tokens error. add_generation_prompt decides whether the trailing assistant opener is included, which it is at inference time. And passing add_special_tokens=False when tokenising an already-rendered template avoids adding a beginning-of-sequence token twice, which the template already inserted.

For hosted models where you cannot render the template, no local method is authoritative. That is not a reason to guess — it is the reason for the next section.

Reconcile instead of estimating

Every response already tells you the true count. The fix is to stop treating your estimate as an answer and start treating it as a prediction with a measurable error.

estimated = my_counter(messages, tools)
resp      = client.chat.completions.create(...)
actual    = resp.usage.prompt_tokens

log.info("token_reconciliation", extra={
    "model":     resp.model,          # the RESOLVED model, not the alias
    "estimated": estimated,
    "actual":    actual,
    "ratio":     actual / max(estimated, 1),
    "delta":     actual - estimated,
    "n_msgs":    len(messages),
    "n_tools":   len(tools or []),
    "has_image": any_image(messages),
})

A week of that field answers everything. A ratio that is a stable 1.03 is per-message template overhead and you can model it exactly. A ratio that jumps with n_tools identifies the schemas. A ratio that is stable within a model and different across models means your tokeniser is wrong for one of them. A ratio that changed on a particular day, with no deploy, means the alias moved.

The version that does raise an exception

Everything above is about hosted APIs, where a mismatch is silent. Running weights yourself gives you a louder version of the same bug, and the messages are worth recognising because they name the wrong thing.

• A tokeniser loaded from a different repository than the weights. The vocabularies differ, so token IDs index into an embedding matrix that means something else. On CPU this surfaces as an IndexError on the embedding lookup; on GPU it surfaces as a device-side assertion — a message about an indexing assertion failing inside a CUDA kernel, which points at the kernel rather than at the tokeniser. If output is fluent nonsense instead of an error, the vocabularies happened to be the same size and the mapping is simply wrong.
• Special tokens added without resizing the embeddings. Adding tokens extends the vocabulary beyond the embedding matrix, and the first use of a new ID indexes past the end. The resize has to happen on the model as well as the tokeniser, and the new rows start untrained.
• A mismatched padding or end-of-sequence token. Not an exception at all — generation simply never stops, or stops immediately, because the stopping token the runtime watches for is not the one the model emits. This is the commonest cause of a local model that runs to max_tokens every single time.

# Cheap assertions worth running once at load time
assert len(tok) <= model.get_input_embeddings().weight.shape[0], \
    "tokeniser vocabulary is larger than the embedding matrix"
print("eos:", tok.eos_token, tok.eos_token_id,
      "| model eos:", model.generation_config.eos_token_id)
print("pad:", tok.pad_token, tok.pad_token_id)
# round trip: a mismatch shows up immediately on real text
s = "Ünïcode, code_ids and 数字 123"
assert tok.decode(tok(s).input_ids, skip_special_tokens=True) == s

Load the tokeniser and the weights from the same identifier, pinned to the same revision. Two identifiers that were the same model last month are not a guarantee about this month.

Budgeting with a measured margin

Once you have the distribution of actual / estimated, the safety margin stops being a superstition. Take the 99th percentile of the observed ratio, and size requests against that rather than against the raw estimate.

# Measured, not guessed. Recompute monthly from the reconciliation log.
P99_RATIO = 1.06

def fits(estimated_tokens, window, max_output):
    worst_case = estimated_tokens * P99_RATIO
    return worst_case + max_output + 200 <= window

Three practices make the whole class of bug rare. Count with the tokeniser that belongs to the model you are actually calling, resolved from the same configuration that chooses the model. Never trim to exactly the window — leave the margin above plus the output reservation, which is the second half of context_length_exceeded. And treat any characters-per-token heuristic as a rough sanity check, never as a budget input; counting tokens before sending and token count mismatches cover the counting side in more depth.