Locale Handling Moves in Three Places at a Migration

“Which provider is better at Japanese” is not answerable and is not the question. Your internationalised product breaks at a migration in three specific places, none of which is about model quality.

Three couplings, not one

When a multilingual feature degrades after a provider change, the reflex is to attribute it to the new model being worse at the language. Occasionally that is true and it is never actionable. The three things that are actionable are couplings between your code and the old provider:

• Token counts. Every budget you compute — how much context to include, when to truncate, what a request will cost — was computed with a tokenizer that no longer applies.
• Language selection. The mechanism by which your prompt tells the model which language to answer in, which is prose and therefore model-dependent.
• Output format. Delimiters, stop sequences, JSON field values and length limits, all of which behave differently once the text is not Latin script.

Work through them in that order, because the first is measurable, the second is testable, and the third is where the visible bugs are.

The tokenizer, and why your counts are wrong

Tokenizers are trained artifacts and each provider ships its own. The same string does not produce the same count across two of them, and the gap is much wider for languages written in scripts that were sparsely represented in the tokenizer’s training corpus. English prose tends to run close to one token per short word; text in a script with poor vocabulary coverage can fall towards one token per character, which is a multiple, not a margin. The mechanism and its billing consequences are the subject of the tokenizer language tax; the migration-specific consequence is narrower and easy to state.

If you compute budgets locally with a tokenizer library — the usual arrangement, because a local count is free and an API call is not — that library encodes one provider’s vocabulary. Point the client at a different provider without changing the counting and every derived number is wrong, in a direction that varies by language. The visible failures are a retrieval step that packs too many chunks and overflows the window on exactly the locales with the worst coverage, and a cost estimate that diverges from the invoice for the same subset of your traffic.

Where a provider does not publish a local tokenizer, it generally offers a counting endpoint instead. Anthropic documents POST /v1/messages/count_tokens, which accepts the same messages array — along with tools, images and documents — and returns an object whose single field is input_tokens. It is an extra round trip, which is why the practical arrangement is: call it during evaluation to calibrate, derive a per-language characters-per-token ratio from the results, and use the ratio locally in production with headroom. Re-derive the ratios at every migration; they are the number that moved.

The instruction that names the language

Almost every internationalised prompt has a line like “Respond in {{locale_name}}” interpolated into an otherwise English system prompt. It works, until it does not, and the failure is the model answering in English regardless. This is a well-known standalone failure — see when a prompt’s output-language instruction is ignored — and what a migration adds is that instruction-following strength for this particular instruction is not uniform across models, so a prompt that never needed reinforcement suddenly does.

Three details make the instruction more portable, and all three are cheap:

• Name the language in the language. “Respond in 日本語” anchors more strongly than “Respond in Japanese” written inside a wall of English, because the target script is present in the context rather than merely described.
• Put the instruction last as well as first. A single mention at the top of a long system prompt competes with everything after it; a short restatement immediately before the user turn is the position that survives model changes best.
• Pass the locale as a structured value too. If you use a schema-constrained output, include a language field carrying the BCP 47 tag and constrain it to the expected value. The model then has to commit to the language as data, and your validator can reject a mismatch without any language detection.

Beware the interaction with few-shot examples. If your examples are in English and the instruction says answer in another language, the examples are evidence against the instruction. That conflict is resolved differently by different models, which is precisely the kind of dependency a migration exposes.

The output format contract

The third coupling is where the user-visible bugs come from, because format assumptions written against English text often hold by accident.

Length budgets expressed in characters do not survive: a hundred characters of Latin text and a hundred characters of a logographic script carry very different amounts of information, and a UI card sized for one will look empty or overflow with the other. Budget in the unit the surface actually constrains — rendered width or lines — and validate after generation rather than trusting the prompt.

Stop sequences and delimiters are the other trap. A sentinel like a triple-hyphen or a bracketed marker is a string the model must emit verbatim; when the surrounding text is in another script, models are more likely to emit a full-width variant of a punctuation character, which is a different code point and does not match. The delimiter that always worked in English silently stops terminating generation, and you get a response that runs to the cap — the fault described on the output-length page. Prefer structured output over sentinel parsing wherever the provider supports it, and where you must use a sentinel, normalise full-width punctuation before matching.

Right-to-left content adds a third: text that is correct as a string can be rendered wrongly once it is concatenated with Latin identifiers, and directional marks introduced by the model may not survive your sanitiser. Test the rendered result, not the string.

A per-locale gate you can actually run

The gate is small and it is the only thing that turns any of the above into something you find before your users do.

1. Take ten to twenty real inputs per supported locale, redacted, from production. Real inputs, because synthetic translations of English inputs share English sentence structure and will not exercise the failure.
2. For each, record three assertions rather than an expected output: the response is in the requested language, it parses under your format contract, and it fits your rendered-length budget. All three are checkable without a human judging quality.
3. Record the token count per input from the provider’s own counter, and derive the characters-per-token ratio per locale. Store it; this is what your production estimator uses.
4. Run the set against the candidate provider before cutover and fail the migration on any locale where an assertion regresses. A per-locale pass rate is the number to take to the go/no-go decision, and it is far more defensible than an aggregate.
5. Keep it running afterwards on a schedule, as a multilanguage consistency check. The locales that break are always the ones with the fewest users, which means nobody reports them.