Regional Arabic Dialects an AI Model Cannot Reliably Tell Apart

“Which Arabic dialect is this?” is not one question. For some sentences it has a confident answer, for many it has none, and the boundary between those two sets is a property of the sentence rather than of the model.

One language tag, thirty languages

In ISO 639-3, published at iso639-3.sil.org, ara is a macrolanguage, and the varieties inside it have their own individual codes: arb for Modern Standard Arabic, arz Egyptian, apc North Levantine, ajp South Levantine, acm Mesopotamian, afb Gulf, ary Moroccan, aeb Tunisian, arq Algerian, and roughly twenty more. Nearly every system in production collapses all of them into the single tag ar.

That collapse is the first source of confusion, and it happens before any model is involved. Training data labelled ar is a mixture of Modern Standard Arabic — the formal written register used across the entire Arabic-speaking world, which nobody speaks natively — and dialect text from wherever the crawl found it. The model learns a distribution over that mixture. Asked to produce “Arabic” it produces something near the centre of it, which is MSA, for the same reason described in neutral Spanish.

The markers that genuinely separate groups

Dialect identification works when a sentence contains items from the closed classes that differ — interrogatives, negation, future and progressive particles, existentials, and a handful of very high frequency adjectives. These are the discriminative features, and they are worth knowing because they tell you what to put in a prompt when you want a specific dialect out.

gloss        MSA          Egyptian       Levantine      Gulf            Moroccan
what         madha        eh             shu            wesh / shinu    shnu
now          al-aan       dilwa'ti       halla'         al-heen         daba
I want       ureed        3ayez / 3awez  biddi          abi / abgha     bghit
good         jayyid       hilw           mneeh          zein            mzyan
there is     hunaka       fi             fi             fi / aku (IQ)   kayn
future       sa- / sawfa  ha-            rah            b- / raah       ghadi
progressive  —            bi-            bi-            —               ka- / ta-
verbal neg   la / lam     ma...sh        ma...sh        ma (no -sh)     ma...sh
how much     kam          kam / bikam    addeesh        kam / cham      shhal

The single most useful line there is the negation row. Egyptian, Levantine and Maghrebi Arabic all use the circumfix ma…sh around the verb; the Gulf varieties generally negate with ma alone and use mub or mu before nominals. A sentence with a negated verb therefore separates Gulf from the rest almost for free, and a sentence without one loses that discriminator entirely.

Maghrebi is the group a model is most likely to get right, and the reason is lexical distance rather than better data: the Amazigh substrate and French borrowing make Moroccan, Algerian and Tunisian text visibly unlike anything east of Libya. The hard problems are Egyptian against Levantine, which share the negation circumfix and the bi- progressive and have been in constant media contact; the internal splits within Gulf (Kuwaiti, Emirati, Qatari, Bahraini, Najdi, Hijazi); and Iraqi against Gulf, which share al-heen-type adverbs and a good deal of vocabulary.

Why many sentences carry no signal at all

This is the part that gets left out of discussions of dialect identification accuracy, and it is the part that explains the numbers. A large fraction of naturally occurring Arabic sentences contain no dialect-discriminative token whatsoever. Write “the meeting is tomorrow at nine in the main office” and every content word is shared; there is no interrogative, no negation, no future particle, no existential. The sentence is not ambiguous between dialects — it is not in a dialect in any recoverable sense.

A model asked to label such a sentence cannot abstain, because the task as posed has no abstain option, so it emits the prior: usually Egyptian, which has the largest dialect footprint in media and therefore in crawled text. That is not the model confusing two dialects. It is the model answering an unanswerable question, and it will do it with the same confident tone it uses for a sentence full of markers.

Two further things destroy signal. Dialect orthography is not standardised — the same spoken word appears with and without vowel letters, with different consonant choices for sounds MSA lacks, so the surface forms fragment. And a great deal of informal Arabic is written in Latin script with digits for pharyngeals, covered in Arabizi and Latin-script Arabic, which throws away the script-level features an identifier relies on.

What the published shared tasks report

Arabic dialect identification is a well-studied task with a public competitive record, which is a better source than any informal probe. The MADAR corpus (Bouamor and colleagues, LREC 2018) provides parallel sentences across twenty-five city dialects. The NADI shared tasks — Nuanced Arabic Dialect Identification, run at the Arabic NLP workshop from 2020 onward — publish an organisers’ overview paper each year with every participating system’s scores. All of these are in the ACL Anthology.

The consistent shape of those results, across years and across teams, is the thing to take away: region-level identification (Gulf against Levantine against Maghrebi against Nile Basin) is substantially easier than country-level, and country-level macro-F1 for the winning systems remains low in absolute terms — far from the accuracy people assume when they ask a general-purpose model the same question. Read the overview paper for the year and subtask you care about and take the number from there rather than from this page, because the numbers move with each edition.

What to do instead of asking for a label

• Do not use dialect identification as a routing key. If a support ticket is routed to a regional team by a predicted dialect, a dialect-neutral message gets routed by the model’s prior. Route on something you actually know — account country, phone prefix, the interface locale.
• Ask for a marker inventory, not a label. “List the dialect-specific words or particles in this text and say which varieties each is associated with; if there are none, say so” is a question the model can answer honestly, and the empty answer is informative.
• For generation, specify the country and give the markers. “Write this in Emirati Arabic” is much weaker than the same instruction plus a short glossary of the particles above. See generating Egyptian Arabic for the variety with the most training coverage, which is the easiest case.
• Decide whether you want MSA. For formal written output — terms, notices, documentation — MSA is correct across the whole region and is what the model is best at. Dialect is for speech-like text, and asking for it in a contract is a mistake that the model will nonetheless carry out.
• Have the output read by someone from that country. The failure mode is a fluent mixture: Levantine syntax with a Gulf word and an MSA verb. It reads as wrong to a native speaker and as fine to every automated check.