Romanizing Arabic Text for Search and Sorting

Mohammed, Muhammad, Mohamed, Muhammed and Mohammad are the same name, محمد, written five ways. If your search index stores one of them, four of your users will not find their own record.

Why there is no single standard

There are at least six published Arabic romanisation systems in current use, each built for a different consumer, and none of them is wrong. ALA-LC is the library standard used by cataloguers in North America. DIN 31635 is the German academic standard and the one most Arabists write in. ISO 233 is a strict letter-for-letter transliteration designed to be reversible. BGN/PCGN, issued jointly by the United States Board on Geographic Names and the Permanent Committee on Geographical Names for British Official Use, is built for maps and drops diacritics readers will not understand. UNGEGN maintains its own for geographical names. Buckwalter is an ASCII-only bijective scheme used in computational linguistics.

Then there is the one with the most users and no standards body at all: Arabizi, the chat alphabet, where numerals stand in for letters that have no Latin equivalent — 3 for ع, 7 for ح, 5 or kh for خ, 2 for hamza, 9 for ص. It is what people actually type on a phone, so it is what arrives in your search box. Arabizi is a script in its own right rather than a corruption of one.

A language model has read all of these. Asked to “romanise this Arabic name” it will produce something reasonable and unlabelled, drawn towards whichever convention dominates the surrounding context — academic-looking input pulls towards DIN-style diacritics, a list of passenger names pulls towards passport spellings. That is not a defect to be fixed. It is the state of the world, and an index has to be built for it.

What the systems actually disagree about

The disagreements are systematic, which is what makes them tractable. Five features account for nearly all the variation.

• Short vowels are not written. Arabic script is an abjad: كتب is the consonant skeleton k-t-b, and it is kataba (he wrote), kutiba (it was written) or kutub (books) depending on vowels that are not on the page. Any romanisation has to supply them from context, which means any romanisation is an interpretation.
• The definite article assimilates. الشمس is written with ال but pronounced ash-shams. Systems that transliterate letters give al-shams; systems that transcribe sound give ash-shams. Both appear in the same corpus constantly.
• Emphatic consonants need diacritics. س and ص are different letters that both become s without a dot below; likewise ت/ط, د/ض, ذ/ظ, ه/ح. Dropping the diacritic merges pairs that distinguish words.
• Hamza and ayn have no Latin letter. ء and ع become ʾ and ʿ in academic systems, an apostrophe in some, a digit in Arabizi, and nothing at all in passports. This is why عمر and أمر flatten together as amr/omar.
• Ta marbuta is context-dependent. The final ة is a in isolation and at in a construct phrase, so مدينة is madina alone and madinat in مدينة نصر.

What to store in the index

The design that works treats every Latin form as derived and never canonical. Three fields, populated once at write time.

• The Arabic original, normalised. This is the only key. Normalise before storing: strip the tashkil (the vowel diacritics U+064B–U+0652), unify the alef forms أ إ آ ا to bare ا, unify ى and ي, unify ة and ه if your data warrants it, and remove the tatweel U+0640 used to stretch a word for typographic reasons. Two spellings of one name will otherwise not compare equal even though both are Arabic.
• One display romanisation, with the system named. Store which standard produced it in a sibling column. A romanised string whose system is unrecorded cannot be regenerated or compared with anything.
• A folded match key. Lowercase, diacritics stripped, apostrophes removed, doubled consonants collapsed, and the vowels reduced or removed entirely. This is the field the query hits.

The folded key is where the Mohammed/Muhammad problem is actually solved, and it is solved by throwing information away on purpose. Fold both to mhmd — consonants only — and they collide, which is what you want. This is the same idea as a phonetic key like Soundex, applied to the fact that Arabic writing is already a consonant skeleton. Store the folded key alongside the full one and search both, ranking exact matches above folded ones.

Matching a romanised query

A user typing into your search box may send you Arabic script, an academic romanisation, a passport spelling, or Arabizi. Detect which before you decide what to do with it: a query containing digits 3, 7 or 2 adjacent to letters is almost certainly Arabizi, not a number.

Then fold the query with the same function you used on the index. This is the part most often got wrong — a query normalised by a different code path than the documents will miss records that are present, and the failure is invisible because the search returns results, just not the right ones. The general shape of that bug is covered in why a transliterated query misses a document it should match.

Do not romanise the document at query time. It is slow, it is non-deterministic if a model is doing it, and it makes the index unreproducible. Romanise once at write time, store the result, and treat the write-time system as part of your schema.

Sorting has a separate answer. Latin alphabetical order over a romanised field is not Arabic alphabetical order, and neither is code-point order over the Arabic. If the sort is user-facing, sort with a Unicode collation for the Arabic locale over the original field; if it only has to be stable, sort the folded key and document that the order is arbitrary. The related trap for Latin-script names with diacritics is in why accented names sort into the wrong place.

What the model is actually for here

Given all of the above, the useful jobs for a model are narrow and it is worth being explicit about them, because the tempting job — “be the romanisation engine” — is the one it is worst at, since it cannot be deterministic and the index needs determinism.

• Supplying vowels from context. This genuinely requires understanding, and a rule-based transliterator cannot do it. Given a sentence, a model can tell you that كتب here is kutub and not kataba.
• Deciding article assimilation. Whether to write al- or ash- depends on the following letter, which is mechanical, but whether the article is present as an article at all sometimes is not.
• Reading Arabizi back into Arabic script. Highly context-dependent, no lookup table, and the direction where a model earns its cost.
• Generating query expansions offline. Ask for the eight most common Latin spellings of a name once, store them as aliases, and never call the model on the query path again.