Why Output Language Sometimes Ignores an Explicit Instruction

The instruction says Answer in English. The user wrote in English. The answer comes back in German. This is a different failure from a model following the user’s language, it has a different cause, and the usual fixes for language drift do not touch it.

A reproduction

The smallest version needs a retrieval stage and a corpus that is not in the output language. Build a context in this order and the failure is reliable rather than intermittent:

system: "You are a support assistant. Always answer in English."

user:
  Context documents:
  ---
  Die Erstattung wird innerhalb von fünf bis sieben Werktagen auf das
  ursprüngliche Zahlungsmittel zurückgebucht. Bei Zahlungen per Lastschrift
  kann die Gutschrift bis zu zehn Werktage dauern. … (2,000 more tokens)
  ---
  Question: How long does a refund take?

A meaningful share of responses come back in German, and the share rises with the length of the German block and falls when the question is moved after it. Nothing is misconfigured. The instruction is a dozen tokens at the very start of a context that is now two thousand tokens of German, ending just before the model begins to generate.

Notice what this rules out. The user’s message is English, so the usual explanation — the model mirrors the user — does not apply. The instruction exists and is unambiguous. Turning temperature down does not help. If your drift disappears when you empty the retrieved context, this is the mechanism you have.

Four sources that outweigh the instruction

• Retrieved documents. The common case. A corpus in one language, a product in another, and a RAG prompt that pastes the documents in without labelling what they are for. The larger the top-k and the longer the chunks, the stronger the pull.
• Prior assistant turns. The strongest source of all. A model continuing its own previous output is the most reliable behaviour it has, so one drifted turn in the history reproduces itself indefinitely. If the drift started three turns ago and every turn since has been wrong, the history is now the cause and fixing the prompt changes nothing.
• Few-shot examples. Examples are a demonstration of the output format, and the model reads the language of the demonstrated output as part of that format. Examples whose outputs are in the wrong language are an instruction, and a more concrete one than your sentence.
• Tool results. A search tool that returns snippets, a database row containing a localised product description, an OCR stage returning text from a scanned document. These arrive late in the context — usually the last thing before generation resumes — which is the worst possible position.

A useful diagnostic: bisect the context. Remove each source in turn and rerun. The one whose removal fixes it is the one to label, and it is rarely the one people assume.

Why a document beats an instruction

Nothing in a context window is marked as an instruction. There is no field the model reads as authoritative and no separate channel for policy. The whole context is one token sequence, and generation continues it. A system role is a convention applied during post-training, and it is a strong convention, but it is competing against statistics rather than overriding them.

Text in a language is overwhelmingly followed by text in the same language — one of the most reliable regularities in any training corpus. Two thousand tokens of German is two thousand pieces of evidence for a German continuation, all of them adjacent to the generation point. Twelve tokens saying “answer in English” is a small correction applied against that, from a long way away. The surprise is not that it fails sometimes; it is that it works as often as it does.

This also explains the partial failures people find confusing. The model answers in English but keeps German terms, headings or units. It is not half-following the instruction — it has resolved two competing signals and landed between them, which is the expected result of combining distributions.

Demoting the context to data

Every fix is a version of the same idea: make the other-language text read as material to be used rather than as text to be continued.

1. Label the block, before and after. A single line before the documents and a single line after is measurably better than one before, because the second one sits between the documents and the generation point. The documents above are reference material and may be in any language. Do not imitate their language.
2. Put the instruction last. Order the prompt as documents, then question, then output-language directive. Never documents last.
3. Delimit clearly. A consistent fence around the retrieved text — an XML-style tag, or a repeated marker — separates it from the conversation. This helps for the same reason a label does: it marks a boundary the continuation should not cross.
4. Use a structured output with an explicit language field. Requiring the model to emit "language": "en" as the first field before the answer makes the language decision an explicit token early in generation, and the answer that follows is conditioned on it. This also gives you something to assert on.
5. Repair the history when the language changes. If a conversation switches output language mid-thread — because the user switched, or because a drift was corrected — rewrite or drop the earlier assistant turns. Leaving them is leaving the cause in place.
6. Translate the retrieved chunks at index time. The heavy option, and sometimes the right one. If your corpus is one language and your users are another, storing a translated field alongside the original removes the conflict entirely, at the cost of translation quality and storage. The retrieval side of that decision is in retrieval over a mixed-language corpus.

Then verify. Detect the language of the output, compare to the request, retry once with stronger reinforcement, and log the rate by source so you can see which of the four is responsible. The retry loop and the detection caveats are in forcing a model to always respond in one language.

The reasoning-trace case

One variant deserves its own note because it looks like a different bug. With a model that produces a reasoning trace before its answer, the trace itself can switch language — most often to the language of the retrieved documents, sometimes to whatever language the model finds most natural for the content. The final answer is then conditioned on a trace in that language, which drags it along.

The symptom is an answer that drifts despite a directive that is positioned correctly, and the tell is that the drift correlates with longer reasoning rather than with longer documents. Where the provider exposes control over the reasoning language, set it. Where it does not, the workable pattern is to separate the steps: one call that reasons and returns a structured intermediate result, and a second call that renders the answer in the target language from that result. Two calls cost more and remove the coupling entirely. The wider version of this problem is chain-of-thought language mismatch.