Why AI Models Refuse More Often in Arabic Than in English

“It refuses more in Arabic” is reported constantly and is two different observations wearing one sentence. One is a model declining requests that are perfectly ordinary. The other is a model complying with requests it would decline in English. Both are documented, they are not contradictory, and telling them apart is the whole of the diagnosis.

Two opposite claims, one sentence

The first claim is over-refusal: a benign Arabic prompt gets declined where its English translation does not. A request to summarise a news report about a conflict, translate a passage of classical poetry, or explain a religious practice comes back as a refusal. The term of art from the English literature is exaggerated safety, and the reference dataset for it — XSTest, published by Röttger and colleagues in 2024 as a set of 250 safe prompts that models should not refuse — is English-only, which is itself part of the story.

The second claim is under-refusal: a harmful Arabic prompt gets answered where its English translation is declined. This is the safety-gap direction, and it is the one the multilingual jailbreak literature measures.

These coexist because the refusal boundary in a language with little alignment data is not shifted, it is noisy. Prompts land on either side of it less predictably than they do in English. A user who writes ordinary Arabic notices the over-refusals; a red team notices the under-refusals; both are describing the same underlying property. The mechanism behind it is the subject of English-centric safety training and its blind spots.

What the published evaluations cover

Arabic appears in a small number of public multilingual safety evaluations, and it is worth knowing exactly what each one is, because the scope determines what can honestly be concluded.

• MultiJail, from Deng and colleagues’ Multilingual Jailbreak Challenges in Large Language Models (2023, later at ICLR 2024): 315 English harmful prompts translated into nine languages, grouped by resource level. Arabic sits in the medium-resource group alongside Korean and Thai. The paper’s headline finding is directional — unsafe-response rates rise as the language’s resource level falls — and that direction, not any single percentage, is what replicates.
• XSafety, from Wang and colleagues’ All Languages Matter: On the Multilingual Safety of LLMs (2023): fourteen safety scenarios across ten widely spoken languages, roughly 28,000 items in total. Its finding is that every model tested was less safe in non-English languages than in English.
• Aya red-teaming, released by Cohere For AI with The Multilingual Alignment Prism (2024): human-written and human-annotated harmful prompts in eight languages including Arabic, rather than machine translations of English ones. That distinction matters more than the sample size.

Notice what is absent. None of these measures over-refusal on benign Arabic prompts, because there is no widely used Arabic counterpart to XSTest. The claim most Arabic-speaking users make about production models is the one the public literature covers least.

Why there is no single number to quote

It is tempting to want “Arabic refusal rate is N% higher”. No such figure is meaningful for four reasons, and each one is a reason to distrust any source that offers one.

• It is per model and per version. Refusal behaviour is set by alignment, which changes between checkpoints of the same model name. A gap measured on one snapshot says nothing about the next.
• It is per prompt set. A refusal rate is a property of the joint distribution of the prompts and the model. Change the prompts from translated English red-team items to natural Arabic user traffic and the number moves by more than the model difference you are trying to detect.
• It depends on what counts as a refusal. A hard refusal, a hedged partial answer, a request to rephrase, and a filter block that never reached the model are four different events. Studies that count only the first understate; studies that count hedging overstate.
• Provider filters sit in front. Some of what users experience as model refusal is a separate classifier, with its own language coverage, returning a policy error before generation starts.

Measuring it on your own prompts

The measurement worth doing is a paired one on prompts from your own product. The design is simple and the discipline is in the pairing: every item exists in both languages and is judged by the same rule.

type Item = { id: string; en: string; ar: string; benign: boolean };

// One request per (item, language, trial). Keep trials >= 5 at temperature 0
// so you measure the model, not the sampler.
async function trial(model: string, text: string) {
  const res = await fetch("https://api.example.com/v1/chat/completions", {
    method: "POST",
    headers: { "content-type": "application/json", authorization: KEY },
    body: JSON.stringify({
      model,
      temperature: 0,
      max_tokens: 300,
      messages: [{ role: "user", content: text }],
    }),
  });
  if (res.status === 400) return { outcome: "filtered" };      // policy error
  const json = await res.json();
  const choice = json.choices[0];
  if (choice.finish_reason === "content_filter") return { outcome: "filtered" };
  return { outcome: "generated", text: choice.message.content };
}

Then classify each generated response as complied, refused or hedged — with a rubric applied by a human on a sample, and a classifier on the rest that you have checked against that sample. Report four rates, not one: over-refusal on benign items in each language, and compliance on harmful items in each language. Because the design is paired, the right test for the difference in proportions is McNemar’s test on the discordant pairs rather than a two-sample test, which is what makes the comparison sensitive at sample sizes you can afford.

The confounds that will ruin the result

Three of these are large enough to produce an entirely spurious gap.

Translation changes the prompt

If your Arabic items are machine translations of English ones, you are partly measuring the translator. Literal translation shifts register, flattens idiom, and occasionally produces a phrase that reads as threatening when the source did not — the effect described in cultural context and false-positive refusals. Have a native speaker adapt the items, and record which ones were changed.

Modern Standard Arabic is not what users write

Almost all Arabic training and evaluation data is Modern Standard Arabic, while most people write a regional variety. A gap measured in MSA is a lower bound on what Egyptian or Gulf users experience, and the varieties differ enough that a model can mishandle one and not another — see Arabic regional dialects and model confusion.

Script and transliteration

Arabic written in Latin letters with digits standing in for letters — Arabizi — is a third condition, and it behaves like neither Arabic nor English. If your traffic contains it, it needs its own arm in the study rather than being folded into the Arabic one.