Why Arabic Semantic Search Misses Obvious Matches

You search an Arabic corpus for a term that appears verbatim in three documents and get none of them back. The model is not confused about the meaning. It is looking at a different sequence of tokens than the one in the index, because Arabic gives you several correct ways to write the same word and a subword tokenizer treats each of them as unrelated.

One word, six legal spellings

Arabic orthography tolerates variation that a Latin-script writer would call a typo. The hamza-carrying alef forms أ إ آ are routinely written as bare ا. Final yeh ي and alef maqsura ى are interchanged constantly, including by native typists and by OCR. Ta marbuta ة is often typed as heh ه. Tatweel, the kashida character U+0640, is inserted purely to stretch a word for justification and carries no meaning at all, so كــتاب and كتاب are the same word with different bytes.

On top of that, text extracted from PDFs frequently arrives in the Arabic Presentation Forms blocks (U+FB50–U+FDFF and U+FE70–U+FEFF) rather than in the standard Arabic block. Those are the positional glyph variants — initial, medial, final, isolated — and a naive extractor emits them one per rendered glyph. The result looks identical on screen and shares almost no code points with the same sentence typed normally.

Every one of those is a different byte string. An embedding model does not see letters; it sees the token ids its tokenizer produced from those bytes. Six spellings of one word are, before the model has done any work at all, six different inputs.

Why a diacritic is not a small edit

The intuition that trips people up is that adding a diacritic is a small perturbation, so the embedding should move a small distance. That intuition comes from character-level thinking, and modern tokenizers are not character-level.

Short vowels in Arabic are written as combining marks — fatha, damma, kasra, sukun and the tanween forms occupy U+064B through U+0652 — and they are usually omitted. When they are present, in scripture, poetry, children’s books or a well-typeset legal document, they sit between the consonants in the code point stream. Arabic characters take two bytes each in UTF-8, and a byte-level BPE tokenizer builds its merges over those byte sequences. Inserting a two-byte combining mark in the middle of a word does not add one token to the word’s token sequence. It interrupts the byte run that the merge table was going to consume, so the merges take a different path from that point on and the word is re-segmented from the insertion onwards.

The consequence is that the vocalised and unvocalised forms of a word can share no subword unit beyond the first letter or two. They are not near-neighbours in the input space, so there is no reason to expect them to be near-neighbours in the output space. Print the token ids for both forms with your model’s own tokenizer before you conclude anything about the model’s Arabic ability; the answer is usually visible there.

Root-and-pattern morphology breaks stem sharing

Subword tokenizers give European languages a useful accident: because those languages build words by concatenation, a frequent stem tends to survive as one token across its inflections, so book and books share a unit and the model gets stem sharing for free.

Arabic does not build words that way. It interleaves a three-consonant root with a vowel-and-affix pattern. From the root k-t-b you get كتب (he wrote), كتاب (book), كاتب (writer), مكتب (office) and مكتبة (library). These are transparently related to a reader and share a root, but the shared material is discontinuous — consonants separated by different letters in each form. A merge table learned from concatenative morphology has no representation for a discontinuous unit. It cannot factor out k-t-b, so it learns each surface form separately, and the rarer forms get split into more, less well-trained pieces.

This is why an Arabic query can fail against a document that a human would call an obvious match. The document says library and the query says books; in English those two share a token, and in Arabic they share a shape the tokenizer cannot see.

Attached particles change the whole word

Arabic writes the definite article and several conjunctions and prepositions attached to the following word. ال (the), و (and), ف (so), ب (with), ل (for) and ك (like) all fuse leftwards. So “and the book” is one orthographic word, والكتاب.

Because BPE merges are greedy from the start of a byte run, a prefix changes the segmentation of everything after it, exactly like a diacritic does. A query for the bare noun and a document containing the prefixed noun do not share the noun’s tokens. Pronoun suffixes do the same thing from the other end. A single Arabic orthographic word can carry a conjunction, a preposition, the article, the stem and a possessive pronoun, and only the middle of that is what the searcher meant.

There is a second layer on top of the orthography: the index is usually Modern Standard Arabic and the query is often dialect. Egyptian, Levantine, Gulf and Maghrebi varieties differ in high-frequency vocabulary and in negation, and a model whose Arabic training data is predominantly MSA news text has thin representations for dialectal forms. That is a real gap, but it is a different gap from the orthographic one, and it is worth separating them before you go shopping for a new model.

The normalization pass, and where it goes wrong

Most of the orthographic damage is recoverable with a deterministic pass applied to documents at index time and to queries at search time. Applying it to only one side is the single most common bug here, and it is invisible: retrieval gets worse rather than erroring.

• Unicode normalize to NFC first. Some Arabic text arrives decomposed; NFC gives you one canonical code point sequence before you do anything else.
• Map presentation forms back to the standard block. NFKC does this, but be aware it does more — it also folds ligatures and converts Arabic-Indic digits, which may or may not be what you want.
• Strip the diacritic range U+064B–U+0652, the superscript alef U+0670, and tatweel U+0640. None of them distinguish words in ordinary prose.
• Unify the variable letters. All hamza-alef forms to bare alef, alef maqsura to yeh, ta marbuta to heh. Pick one direction and apply it everywhere.
• Keep a lexical field alongside the vector. A BM25 index over a lightly stemmed version of the normalized text recovers exact-term matches the embedding loses, and hybrid retrieval is cheaper than a model migration.

Once that pass is in place, re-run the queries that failed. If they still fail, the problem is the morphology and the dialect gap rather than the spelling, and the next lever is a lexical field with a real Arabic stemmer or a reranking pass — not another embedding model. The same ordering applies when the query arrives in Latin letters, which is a separate failure with a separate fix; see why transliterated queries land nowhere near their target.