What a Migration Does to a Prompt's Multi-Language Consistency

The prompt is one English system message that says “reply in the customer’s language”. It worked identically for eight locales for a year. On the new model, Dutch and Portuguese are fine, German has lost its formal register, and about one Thai reply in four comes back in English.

The symptom: even before, ragged after

The reproducible form is simple. Take twenty-five real inputs per locale, run them through both models with the identical prompt, and score three things per output: is the reply in the requested language, is the register correct, and does it satisfy the same format validator you use for English. What you see is not a uniform quality drop. It is two or three locales falling off a cliff while the rest are flat or slightly better, and an aggregate number that barely moves.

That shape is the diagnosis. Uniform degradation is a prompt problem. Ragged degradation by language is three different problems wearing one costume, and treating it as one problem is why the obvious fix — adding “ALWAYS reply in the same language as the user” in capitals — helps a little and then stops.

Three separate causes with three separate fixes

• Language selection is a default, not an instruction. Your prompt is in English and the content is not. Which language the output takes is decided by a competition between the instruction language, the content language, and the model’s own tendency, and that tendency is tuned per model. It leans toward English on many models, and it leans harder when the input is short, when the input contains English proper nouns, or when the requested output is structured. This is the cause behind the reply that comes back in English.
• Training mix is uneven and differs per model. Two models can have comparable aggregate multilingual scores and very different per-language profiles, because the underlying data mix is not the same. Nothing in your prompt can compensate for a language the target model saw less of; you can only detect it and decide. This is the cause behind the collapsed German register and behind subtler damage such as broken diacritics or wrong pluralisation.
• The token budget buys different amounts of text. Tokenizers differ between models and they do not differ uniformly by script. A max-output setting that comfortably fitted a full Thai or Arabic reply on one model can clip it on another, and the clipped output usually fails the JSON validator rather than reporting itself as truncated. See the general treatment of the tokenizer language tax for why the ratio is what it is.

Three causes, three fixes, and only the first two are prompt work.

Pin the language as data

Stop asking the model to infer the output language and stop asking it in prose. Detect the language in code — you almost certainly already do, for routing — pass it as an explicit tag in the request, and require it back in the output where a validator can check it:

// Request side: the tag is an input, not an inference.
const system = [
  "Reply in " + LOCALE_NAMES[locale] + " (" + locale + ").",
  "Set the \"language\" field of your response to exactly \"" + locale + "\".",
  REGISTER_RULES[locale],   // e.g. "Use formal address (Sie), never du."
].join("\n");

// Output schema: the language is a checked field, not a hope.
{
  "type": "object",
  "required": ["language", "body"],
  "properties": {
    "language": { "enum": ["nl-NL", "de-DE", "pt-BR", "th-TH", "ar-EG"] },
    "body": { "type": "string" }
  },
  "additionalProperties": false
}

// Call side: a wrong language is now a rejectable response.
if (out.language !== locale) {
  metrics.inc("language_mismatch", { locale });
  return retryOnce(locale) ?? escalateToHuman();
}

Two things this buys. A drift back to English becomes a counted event with a locale label rather than a customer complaint, and the per-locale REGISTER_RULES line gives you somewhere to put the formality rule that the old model applied from its own defaults. German formal address, Japanese politeness level and Thai pronoun choice are all things a model can do and none of them are things it will keep doing across a migration unless you state them.

The other lever worth spending is per-locale few-shot examples. One correct example in the target language, in the target register, in the target format, does more for a weak locale than three paragraphs of instruction in English — and it is the only mechanism here that addresses the training-mix cause at all. Keep the examples in the same per-locale file as the register rules so they cannot drift apart.

Give each locale its own token budget

A single global output cap is a bug in a multilingual system. Derive the cap per locale from a measurement on the target model:

1. Take a reference response of the length you consider a full answer, professionally translated into each supported locale.
2. Count the tokens of each translation using the target model’s own token counting endpoint, not a third-party tokenizer library and not a character estimate.
3. Set the per-locale cap to the largest of those counts multiplied by a headroom factor — 1.5 is a reasonable starting point — and store it next to the locale’s register rules.
4. Alert on the length-limited stop reason per locale. If one locale produces most of your truncations, its cap is wrong; if all of them do, your headroom factor is.

What to test before rollout

Run these per locale and report per locale. The single most damaging reporting habit in multilingual work is the average: eight locales at 97% and one at 40% averages to 91%, which looks like a mild regression and is in fact one entirely broken market.

• Language fidelity. Proportion where the output language matches the requested locale. Check the body, not just the declared field — a model can set the field correctly and write in English.
• Register. For locales with a formality distinction, a regex or small classifier over the informal pronouns is usually enough to catch a collapse.
• Script integrity. Diacritics, combining characters, and right-to-left marks preserved. Mojibake and stripped accents are common and easy to miss in review by someone who does not read the language.
• Locale formatting. Dates, decimal separators and currency. These are the ones a model most often silently normalises to US conventions when it is under pressure.
• Tool-call arguments. If the model calls tools, check that arguments stay in your canonical language while only the user-facing text is localised. A translated enum value is a runtime error, and it is the failure most likely to reach production because English-language testing never sees it.

Report the per-locale table and the sample size beside it. The method for deciding whether a per-locale drop is real is the same paired test used for format conformance, applied once per locale rather than once overall.