Does System Prompt Language Change Output Quality?

The honest answer has three parts: there is published evidence that instruction language affects results on some tasks in some languages, there is no public benchmark that varies the system prompt alone, and the test that would settle it for your case takes about an hour.

What is documented

Three things can be cited rather than asserted.

The system prompt has a defined, elevated role. Both major API vendors document a hierarchy in which platform or developer instructions take precedence over user messages — OpenAI sets this out in its published model spec, and Anthropic documents the system parameter’s role in its API reference. That is a statement about precedence, not about language, but it establishes that the system prompt is not merely another message and that changes to it are not equivalent to changes elsewhere.

Instruction language affects task performance in the multilingual reasoning literature. The MGSM work at arXiv 2210.03057 compares configurations in which the model is prompted to reason in English against configurations in the problem’s own language, and finds English-mediated prompting competitive or better, with the largest differences on the least-represented languages. Follow-up work on cross-lingual prompting reports the same direction. This is real evidence that which language the instructions are in is not neutral.

Instruction-tuning data is not evenly distributed across languages. This is not controversial and it is visible in the published composition of open instruction datasets: English dominates, a handful of high-resource languages follow, and the long tail is thin. Instruction-following is a learned behaviour, so its reliability tracks that distribution.

What is not documented

Nothing in the public record isolates the system prompt as the single varied factor. The multilingual prompting papers vary the whole prompt, usually including the examples and often the problem statement itself. They answer “does prompting in English help”, which is a different and broader question than “holding the user message, the examples and the sampling parameters constant, does moving only the system prompt into the target language change the output”.

So there is no figure to quote here, and this page does not invent one. If somebody offers you a percentage for this effect, ask what varied. In most informal write-ups the answer is: everything.

Why it could matter at all

Three mechanisms predict an effect, and they point in different directions, which is why the question does not have an obvious answer.

• Instruction-following reliability follows the tuning data. If constraint adherence was mostly trained in English, an English system prompt is more likely to be obeyed — the constraint honoured, the format respected, the refusal rule applied. This predicts English wins on compliance.
• The prompt sets the language context. Every token in the context conditions the next one. A system prompt in Portuguese biases the continuation toward Portuguese before the user message arrives, which predicts fewer language-drift failures. This predicts native wins on output language.
• Register transfers from prompt to output. A native system prompt written by a native speaker carries register and terminology that the output will echo, for the reason argued in writing prompts natively. This predicts native wins on fluency — and predicts that a translated native system prompt is the worst of the three options, because it gets the drawback without the benefit.

Those predictions are compatible: English instructions plus native examples plus an explicit output-language constraint is the configuration all three mechanisms favour. That is the configuration worth testing against, not English-only and not native-only.

A design that isolates the variable

Four arms, one varied factor, everything else byte-identical.

Fixed across all arms:
  model + version         pinned, recorded
  temperature             0 (or a fixed seed if the API offers one)
  user message            identical bytes, in the target language
  few-shot examples       identical, native, in the constant prefix
  output schema           identical
  inputs                  the same 100 real items from your traffic

Varied (system prompt only):
  A  English system prompt
  B  Native system prompt, written by a native speaker
  C  Native system prompt, machine-translated from A
  D  English system prompt + explicit "respond only in <lang>"

Scored separately, never as one number:
  1  output language correct           automatic: script/Unicode check
  2  schema valid                      automatic: parse + validate
  3  constraints honoured              automatic: length, banned terms,
                                       required sections present
  4  task correct                      human or a rubric, on a sample
  5  fluency / register                native speaker, blind to arm

Three details decide whether the result means anything.

• Arm C is not padding. It separates “the prompt is in the target language” from “the prompt was written by someone who speaks it”. Most informal comparisons conflate these and then attribute the result to the language.
• Score the five outcomes separately. They routinely move in opposite directions — an arm that produces the most natural prose may be the one that most often ignores the schema. A single blended score hides exactly the trade-off you are trying to observe.
• Blind the human rater to the arm. Whoever set the experiment up has a hypothesis, and fluency judgements are the easiest thing in this list to bias.

A hundred items is enough to see a large effect and not enough to resolve a small one. If the arms come out close, the correct conclusion is that the effect is small on your task, not that you need a tie-breaker. Wire the whole thing into the harness described in testing prompt consistency across languages so it re-runs when you change model.

Acting on the result

• If the arms are close, choose for maintenance. One English system prompt with a language variable is far cheaper to keep correct than forty native ones, and the forty will drift. Language belongs in the examples and in the output constraint, where it costs one short block per locale.
• If English wins on compliance and loses on fluency — the most common shape given the mechanisms above — keep the English machinery and fix fluency with native examples rather than by translating the instructions.
• If a specific language is much worse in every arm, the system prompt is not your problem. That is a coverage problem, and it is addressed in prompting in a low-resource language.
• Re-run on every model change. This is a refresh question by nature; the answer is a property of a checkpoint.