Testing Prompt Output Consistency Across Languages

A suite written entirely in English tests one language and asserts nothing about the others. The fix is not to translate the assertions — it is to identify the parts of the output that should be identical in every language, and assert only on those.

Decide what is allowed to vary

Start by splitting the response into two halves. The prose is allowed to vary: different wording, different length, different idiom. Almost everything else should not. Write the invariant list down before writing a single case, because it is the actual content of the test:

• The schema. Same keys, same types, same nesting. Field names are part of your contract and must not be translated by the model, which is a real failure — ask for JSON in German and a model will sometimes helpfully return "betrag" where your parser wants "amount".
• Enum values. A classification that returns refund_request in English and demande_de_remboursement in French is a bug with a hundred-per-cent English pass rate.
• Numbers and dates as data. The extracted value of total is the same number regardless of whether the input wrote it with a comma or a full stop as the decimal separator. This one catches real extraction bugs.
• Tool selection. Same tool, same arguments — covered in depth by testing translated prompts against tool calls.
• Refusal or non-refusal. A prompt that is refused in English and answered in Indonesian is a safety gap, and it is only visible if both are in the suite.

One case, many languages

Structure the fixture so that the expectation is written once and the inputs are a mapping from language tag to text. This keeps the invariant single-sourced — the failure mode of the naive layout, where each language gets its own copied expectation, is that somebody fixes a French expectation to match observed output and the invariant silently forks.

# fixtures/cases/refund_intent.yaml
id: refund-intent-over-limit
expect:
  schema: schemas/intent.json
  intent: refund_request
  amount: 249.99
  reply_language: same_as_input
inputs:
  en: "I want my money back for order 4471, it was 249.99 dollars."
  de: "Ich möchte mein Geld für Bestellung 4471 zurück, es waren 249,99 Dollar."
  fr: "Je veux être remboursé pour la commande 4471, cela faisait 249,99 dollars."
  ja: "注文4471の返金をお願いします。249.99ドルでした。"
  ar: "أريد استرداد أموالي عن الطلب 4471، كان المبلغ 249.99 دولارًا."

# tests/multilingual/test_invariants.py
import json, pytest
from jsonschema import validate
from app.pipeline import classify

CASE = load_case("refund_intent")

@pytest.mark.parametrize("lang", sorted(CASE["inputs"]))
def test_invariants_hold_in_every_language(lang):
    out = classify(CASE["inputs"][lang])
    validate(instance=out, schema=load_schema(CASE["expect"]["schema"]))
    assert out["intent"] == CASE["expect"]["intent"]
    assert out["amount"] == pytest.approx(CASE["expect"]["amount"], abs=0.01)

Note the German and French inputs use a comma as the decimal separator. That is not decoration; it is the case. An extractor that strips non-digits will turn 249,99 into 24999, and the English-only suite will never show it. Include one thousands-separator case for the same reason: 1.249,99 and 1,249.99 are the same amount.

Assert the reply’s own language

The single most common multilingual failure is a model that answers in English regardless of the input, usually because the system prompt is in English and outnumbers the user turn. This deserves its own assertion rather than being folded into a quality score, because it is binary, cheap to check, and the one users notice immediately.

Detect the language of the free-text field with a library rather than by heuristic, and assert it matches the input’s tag. Two practical cautions. Short strings are unreliable for every detector, so apply the assertion only to fields above a length threshold and say so in the test. And languages that share a script — Spanish and Portuguese, Indonesian and Malay — produce genuine detector confusion, so treat those as a set rather than a single expected value instead of leaving a case that fails one run in ten.

Where an instruction to reply in the user’s language is being ignored outright, the mechanism is worth reading separately in the page on that specific failure.

Your translations are part of the test

If the non-English inputs came out of machine translation, a failure is ambiguous: it might be the model, or it might be that your French input is not what a French speaker would write. Machine-translated test data also tends to be unnaturally literal, which makes it easier than real user text — it preserves English syntax, so a model that only really handles English syntax passes.

So: have a speaker review the inputs at least once, keep the reviewed version in the repository as the fixture, and treat changing it as a reviewed change rather than a regeneration. Record who reviewed each language and when, in the fixture file. When a case fails a year later, the first question will be whether the input is right, and a name and a date answer it in seconds.

Include at least one right-to-left language and one non-Latin script even if you do not officially support them, as a canary. They surface encoding and truncation bugs in your own pipeline — not in the model — that will eventually reach you through a user pasting text you did not plan for.

Reading a failure that is only in one language

When one language fails and the others pass, work through the layers in order rather than reaching for the prompt. Check the raw bytes first: a mojibake sequence in the logged prompt means the failure is your encoding, not the model’s comprehension. Check the token count next — the same sentence in a non-Latin script can consume several times the tokens of its English equivalent, and a case that only fails in Japanese is often a case that only truncates in Japanese. Then check whether the system prompt language is fighting the user turn.

Only after those three does the interesting hypothesis apply, which is that the model genuinely handles the task less well in that language. That one is not fixable by a test, but it is extremely fixable by a routing decision, and knowing which of your supported languages sit below the line is the reason to run this suite at all.

Keep one further column in the report: the token count of the rendered prompt per language. It costs nothing to record, it explains a large share of the failures above without any further investigation, and it is also the number your finance team will ask about when the same product costs noticeably more per conversation in one market than another. A suite that already runs every case in five languages is the cheapest place you will ever get that measurement, and it turns a consistency test into an input for a pricing and routing decision as well.