Normalizing Right-to-Left Text Before Search Indexing
10 min read · updated August 11, 2026
Arabic and Hebrew both write the same word several legal ways, and none of the differences is visible to a reader skimming the page. A search index built on the raw text has a separate posting list for each variant, so a query matches whichever spelling the writer happened to use and nothing else.
The order of the pipeline
Five stages, and swapping any two of them changes the result. The reason order matters is that each stage assumes the previous one has already collapsed a class of variation: unifying letters before folding presentation forms means the presentation forms escape the unification, and stripping marks before compatibility decomposition means the marks inside presentation-form ligatures survive.
- 1. NFKC. Folds the Arabic Presentation Forms blocks (U+FB50–U+FDFF and U+FE70–U+FEFF) and Hebrew presentation forms (U+FB1D–U+FB4F) onto normal letters. Text extracted from PDFs arrives in presentation forms routinely, because that is what the font encoding contained.
- 2. Remove format characters. Bidi controls, tatweel, soft hyphens. These have no lexical content.
- 3. Strip diacritics. Arabic tashkeel, Hebrew niqqud and cantillation.
- 4. Unify letter variants. Alef forms, teh marbuta, alef maksura, Hebrew final forms, Persian versus Arabic kaf and yeh.
- 5. Fold digits. Arabic-Indic and extended Arabic-Indic to ASCII.
The whole pipeline runs on the query too, identically, from the same function. An asymmetric pipeline is worse than no pipeline: it turns a partial match problem into a total one.
Arabic: marks, alef and hamza
Arabic short vowels are written with marks above and below the consonants, and in ordinary prose they are simply omitted. Religious texts, poetry, dictionaries and children’s books include them. The same word therefore appears both with and without, and the marked form has extra code points that a tokeniser treats as part of the token.
Arabic marks to remove U+064B ً fathatan U+064C ٌ dammatan U+064D ٍ kasratan U+064E َ fatha U+064F ُ damma U+0650 ِ kasra U+0651 ّ shadda U+0652 ْ sukun U+0670 ٰ superscript alef U+0640 ـ tatweel (kashida) — a stretching character, not a letter Letter variants to unify U+0622 آ ┐ U+0623 أ ├→ U+0627 ا alef U+0625 إ ┘ U+0671 ٱ ┘ U+0629 ة → U+0647 ه teh marbuta → heh U+0649 ى → U+064A ي alef maksura → yeh Persian-vs-Arabic pairs (pick one target per index) U+06A9 ک ↔ U+0643 ك keheh / kaf U+06CC ی ↔ U+064A ي farsi yeh / yeh
The alef unification is the one that earns its place. The hamza on an alef marks a glottal stop and its position depends on grammatical context, so the same word is spelled with U+0623 in one sentence and U+0627 in another, and a very large share of written Arabic simply omits the hamza altogether. Treating the four alef forms as distinct letters at index time means a query for a common word misses most of the corpus.
Teh marbuta is a judgement call rather than a clear win. It is the feminine ending, pronounced as h or t depending on context and often written as plain heh informally. Mapping it to heh increases recall and merges some genuinely different words; leaving it alone splits every feminine noun across two forms. Most Arabic analysers, including Lucene’s, map it. Decide once and record why.
The Persian pairs are not optional if your corpus mixes languages. Persian keyboards emit U+06A9 and U+06CC; Arabic keyboards emit U+0643 and U+064A. Visually the difference is a pair of dots in the final position, which no reader notices, and untreated it splits every Persian document containing a common letter into a separate index space from every Arabic one.
Hebrew: niqqud and final forms
Hebrew has the same vowel-marking situation and one extra problem of its own.
Hebrew marks to remove U+0591–U+05AF cantillation marks (te'amim) U+05B0–U+05BC niqqud (vowel points), incl. dagesh U+05BC U+05BD meteg U+05BF rafe U+05C1, U+05C2 shin dot, sin dot U+05C7 qamats qatan Final forms — five letters change shape word-finally U+05DA ך → U+05DB כ kaf U+05DD ם → U+05DE מ mem U+05DF ן → U+05E0 נ nun U+05E3 ף → U+05E4 פ pe U+05E5 ץ → U+05E6 צ tsadi Punctuation that is not ASCII U+05BE ־ maqaf (Hebrew hyphen) U+05F3 ׳ geresh — often typed as ASCII ' U+05F4 ״ gershayim — often typed as ASCII "
The final forms matter because Hebrew attaches prefixes. A word ending in mem is written with the final form when it stands alone, and the same root can appear followed by a suffix, at which point the letter reverts to its medial shape. Two different code points for what is grammatically one letter means the same lemma indexes twice.
Folding final to medial is the conventional direction, and it is lossy: מים ends in a final mem and a hypothetical medial-mem spelling would be a different string but the same word, so the fold gains recall and loses nothing a searcher cares about. Doing it at index time only, and not on the query, is the mistake to avoid.
Geresh and gershayim deserve special handling because they mark acronyms and abbreviations, which are exactly the high-value tokens in a news or legal corpus. Normalising them to ASCII apostrophe and quote — or to nothing — is fine, as long as the query gets the same treatment, and as long as your tokeniser does not then split the acronym at that character.
Bidi controls, ZWNJ and digits
Bidirectional text carries invisible characters that exist purely to instruct the rendering algorithm. They are U+200E LEFT-TO-RIGHT MARK, U+200F RIGHT-TO-LEFT MARK, the embedding and override controls U+202A–U+202E, and the isolates U+2066–U+2069. None carries meaning. All of them will end up inside a token if you leave them in, which is how a word copied out of a rendered web page fails to match the same word typed into a search box.
The zero-width non-joiner, U+200C, is the exception that must not be deleted blindly. In Persian it separates morphemes inside a written word: the present continuous prefix and the verb are one orthographic unit joined by a ZWNJ. Deleting it glues two morphemes into a string that is not a word; replacing it with a space splits them into two tokens, which is what Lucene’s Persian character filter does and is usually the better default for retrieval. Zero-width joiner, U+200D, is different again and is load-bearing in emoji sequences, so a global strip is not safe on mixed content.
Digits are the easy win. Arabic-Indic digits occupy U+0660–U+0669 and the extended (Persian and Urdu) set occupies U+06F0–U+06F9. A document written with either indexes no numeric token that an ASCII query will find. Fold to ASCII in the index and fold the query the same way, and be aware you have just made the display form unrecoverable from the key, so keep the original field for showing.
The pipeline
- Write it as one pure function that takes a string and returns a string, with no configuration read at call time. Export it from one module.
- Order the stages as above: NFKC, remove format characters, strip marks, unify letters, fold digits. Add a unit test per stage that asserts the intermediate value, so a reordering is caught rather than merely observed later as bad recall.
- Decide the two judgement calls explicitly and record them in a comment: teh marbuta to heh or not, and Persian letters folded to Arabic or Arabic to Persian. Both are corpus-dependent and neither has a universally right answer.
- Apply it at ingest, before chunking, so that chunk boundaries are computed on the normalised text. Applying it after chunking means the offsets you stored point into a different string.
- Apply the identical function to the query in the same request path. If the two live in different services, ship the function as a shared library with a version number rather than reimplementing it.
- Keep the raw text in a separate field for display and highlighting. The normalised form is a key, not content, and showing a reader Arabic with the hamza removed is a visible quality regression.
- Verify with a hexdump, not with your eyes. Every difference in this pipeline is invisible on screen by construction.