Testing Whether a Prompt Works the Same Across Languages

The evaluation everybody builds first reports one number per language and an average across them. It is the least useful shape available: a language scoring badly could be a prompt problem, a model problem, a test-case problem or a rubric problem, and one number cannot tell you which. The fix is a control condition, and it costs one extra run per case.

The question the suite has to answer

You are not trying to find out how good the model is at Turkish. You are trying to find out whether your prompt behaves the same way in Turkish as it does in English. Those are different questions and they need different arithmetic.

A case that fails in Turkish and also fails in English is a task failure — your prompt is wrong, or the case is hard, and language is not involved. A case that passes in English and fails in Turkish is a language failure, and only those belong in the multilingual bug queue. Without the English run you cannot tell them apart, which is why most multilingual evaluation ends with a list of failures nobody can act on.

So every case is run twice: once in the target language and once in English with the same content. The English run is the control. The statistic you care about is the gap between them, per language, per check.

Building the case set

Cases have to be parallel in content and native in form, and those two requirements pull against each other. The compromise that works: write the case in English, have it rewritten natively rather than translated in each language, and record which cases were rewritten and which were machine-translated. Machine-translated cases are usable — they are far better than no coverage — but they carry a known bias, because translationese input is easier for a model to handle than real user text in the same language, and a suite built entirely on it reports scores that are too high.

Label each case with the property it is testing, not just its language. Useful labels, each of which corresponds to a real failure class:

• Plain. An ordinary request. Baseline.
• Register-sensitive. A case where a formal and an informal answer are both fluent but only one is correct for your product. In German or Japanese this is checkable mechanically.
• Constraint-sensitive. A case where the prompt’s negative constraint has to hold — no promise of a date, no invented reference number. Constraints are where translated instructions fail first.
• Adversarial for the language. Code-switched input, a proper noun in another script, a number in a locale format, an ambiguous pronoun. One per known trap.
• Refusal. Something the assistant must decline. Refusal behaviour is well known to differ by language, and it is the failure with the largest downside.

Twenty cases per language covering those five labels is far more informative than two hundred plain ones, because the plain cases nearly all pass and tell you nothing.

Deterministic checks before any judge

Run the cheap, exact checks first. They catch most real regressions, they cost nothing, they never disagree with themselves, and unlike a model-based judge they do not have their own per-language quality problem — a judge model scoring Amharic is subject to exactly the weakness you are trying to measure.

• Output language. Detect the language of the generated text and compare against the requested tag. This is the single highest-yield check in the suite.
• Schema validity. Parse the JSON, check the enum values against the English identifiers, check required fields are non-empty. Localised enum values are a common and silent break.
• Glossary. Assert that every do-not-translate term appears verbatim, and that its known bad translations do not.
• Register. For languages with grammaticalised formality this is a regex. German formal: no standalone du, dich, dir, dein. Japanese polite: the sentence-final forms are です and ます, so a sentence ending in だ is a violation.
• Length ratio. Compare output length against the English control, in characters and in tokens separately. A ratio far outside the range you expect usually means truncation at max_tokens rather than a style difference, and truncation in a language with an expensive tokenizer is the most common language-specific bug that looks like a quality problem.
• Script. For languages written in more than one script, assert the code-point ranges. A Simplified Chinese request answered in Traditional characters passes a language detector and fails a user.

Only after those pass is a rubric judge worth running, and it should score the same rubric on the target-language output and the English control, so the comparison is between two judgements rather than against an absolute scale.

The harness

type Case = {
  id: string;
  label: "plain" | "register" | "constraint" | "adversarial" | "refusal";
  byLanguage: Record<string, string>;  // tag -> input text; "en" is the control
  nativeFor: string[];                 // tags where the text was written, not translated
};

const CHECKS = [
  outputLanguageCheck,   // detected tag === requested tag
  schemaCheck,           // JSON parses, enums are the English identifiers
  glossaryCheck,         // do-not-translate terms present, bad variants absent
  registerCheck,         // per-language regex, skipped where not applicable
  lengthRatioCheck,      // vs the English control, characters and tokens
  scriptCheck,           // code-point ranges for zh-Hans / zh-Hant / sr-Cyrl …
];

async function run(cases: Case[], tags: string[], model: string) {
  const rows = [];
  for (const c of cases) {
    const control = await ask(buildPrompt(profiles.en, c.byLanguage.en), model);
    for (const tag of tags) {
      if (tag === "en") continue;
      const input = c.byLanguage[tag];
      if (!input) continue;                       // record the gap, do not silently skip
      const out = await ask(buildPrompt(profiles[tag], input), model);
      for (const check of CHECKS) {
        const target = check(out, { tag, control });
        const base = check(control, { tag: "en", control });
        rows.push({
          case: c.id, label: c.label, tag, model,
          check: check.name,
          pass: target.pass,
          controlPass: base.pass,
          // the number that matters: failed here but passed in English
          languageGap: base.pass && !target.pass,
          native: c.nativeFor.includes(tag),
        });
      }
    }
  }
  return rows;
}

Keep the run at temperature zero if the provider offers it, and run each case more than once anyway if you can afford to. Language drift in particular is not deterministic even at low temperature, and a single run reports it as a coin flip rather than as a rate.

Reading the results

Pivot on languageGap, grouped by tag and check. That table is the deliverable, and it is readable in a way an average is not: it says that Turkish fails the register check on eight of twenty cases while passing every other check, or that Thai fails only the length-ratio check, which means you have a max_tokens problem and not a prompt problem.

Two things not to do with the numbers. Do not average across languages — a mean over twelve languages hides the one that is broken, which is the only one you were looking for. And do not compare the raw pass rate of a language with many native cases against one with only translated cases; the second is inflated, and the native flag is in the output rows so you can split them.

Report coverage explicitly too. A language with no cases for a label scores nothing rather than scoring well, and the silently-skipped case in the loop above is why the code records the gap instead of dropping it. The broader version of this problem — languages with no evaluation set at all — is the subject of benchmark coverage gaps across languages.