Why Transliterated Queries Fail Semantic Search

A large share of Hindi, Arabic, Greek and Russian speakers type their own language in Latin letters, because that is what the keyboard does. Your index is in the native script. The embedding of the query does not land near the document; it lands near English text about nothing in particular.

One query, two scripts

The user wants to know where the book is. Two ways to type it:

Native:      किताब कहाँ है
Romanized:   kitab kahan hai

Same words. Same meaning. Zero shared bytes.

There is no character-level relationship between the two strings that a model could exploit. Devanagari occupies U+0900–U+097F and encodes to three bytes per character in UTF-8; the romanization is ASCII. Nothing in the byte stream connects them. The only thing that could connect them is a learned association — the model having seen romanized Hindi and Devanagari Hindi used for the same concepts, often enough and clearly enough to place them together.

For most models that association is incidental rather than trained. Romanized Indic text is abundant on social platforms, so a model trained on a broad crawl has seen some; but it has rarely seen it paired with the native script in a way that a contrastive or distillation objective could use. Compare this with a genuine language pair like French and English, where the alignment was trained deliberately on millions of parallel sentences. Romanized Hindi against Devanagari Hindi is, from the model’s point of view, an untrained language pair that happens to involve the same language.

What the tokenizer does with kitab

Now the part that turns a missing association into an active error. The merge table of a byte-level BPE tokenizer trained largely on English has strong opinions about ASCII letter sequences, and it applies them to romanized Hindi.

kitab   →  likely splits around English-frequent units
           "kit" appears in kit, kitchen, kitten, kits
           "ab"  appears in ability, about, table, abs

kahan   →  "ka" / "han"  — both frequent English fragments
hai     →  "hai" or "ha" / "i" — "ha" carries laughter, "hai" is rare

None of these fragments was ever associated with books,
locations, or the Hindi copula. They were associated with
English words that happen to contain the same letters.

So the query’s tokens are not merely uninformative, they are misinformative: each carries whatever meaning it acquired from thousands of English contexts. The pooled vector is a combination of English fragment representations, and it lands in the Latin-script, English-adjacent region of the space.

Add the language-identity component described in why embeddings cluster by language and the outcome is determined. The query is on the English side of a large high-variance direction; the document is on the Devanagari side. The similarity between the romanized query and the correct Devanagari document is frequently no higher than its similarity to an unrelated English document, because the dominant shared feature between query and unrelated English text — Latin script, English-shaped subwords — is stronger than any semantic signal in either comparison.

This is also why the failure is so confusing to debug. The system does not return nothing. It returns plausible English results, or the least bad Devanagari result, with ordinary-looking similarity scores. Nothing in the telemetry says the query was in a script the index does not contain.

There is no standard romanization

A lookup table would solve this if romanization were deterministic. It is not. Formal schemes exist — ISO 15919, IAST and the Hunterian system for Indic scripts — and essentially nobody types them, because they need diacritics that are harder to enter than the native script would be.

किताब   →  kitab · kitaab · kithab · kitāb
कहाँ    →  kahan · kahaan · kaha · kahain
हूँ     →  hun · hoon · hu · huun
नहीं    →  nahi · nahin · nehi · nahee

The variation is not random: it tracks vowel length, nasalisation and the speaker’s regional accent, none of which the Latin alphabet marks. The mapping is many-to-many in both directions, so any table has to be a candidate generator rather than a function. The same holds for Arabizi, where Arabic sounds with no Latin equivalent are written as digits — 3 for ain, 7 for haa, 2 for hamza — a convention no general-purpose tokenizer has any reason to know about. See how models handle Arabizi.

Detecting a romanized query

You cannot fix what you do not notice, and the standard detector will not tell you. Given kitab kahan hai, most language identifiers return English or Indonesian or Somali with unhelpful confidence, because they were trained on native-script text and the script is the strongest feature they have.

Two cheap signals work better than a general detector here:

• A romanized function-word list. Hindi function words are short, extremely frequent and distinctive when romanized: hai, hain, nahi, nahin, kya, aur, mein, ka, ki, ke, ko, se, bhi, kaise. Two hits in a short query is a strong signal, and the list takes an hour to build per language.
• Out-of-vocabulary rate against English. Romanized Hindi is mostly not English words. A query where most tokens are absent from an English wordlist, in Latin script, with a Hindi-speaking user base, is romanized Indic until proven otherwise.

Combine them with what you already know — the user’s locale, their interface language, which corpus languages exist — and treat the result as a flag that triggers extra query paths, not as a routing decision that excludes others.

Fixing it in the query path

1. Transliterate the query into the native script and search with both. Generate two or three candidates rather than one, because the mapping is ambiguous, and fuse the result lists by rank. Deterministic libraries exist for Indic scripts; a small language model does it well too and resolves ambiguity using the rest of the query.
2. Index a romanized rendering of every document as a second vector. This makes the corpus reachable from Latin-script queries directly. Your romanization of the corpus is at least consistent, even though the user’s is not, which is why this works better paired with the next step than alone.
3. Add fuzzy lexical matching over the romanized field. Bounded edit distance or a phonetic key absorbs the vowel-length and nasalisation variation that defeats exact matching. This is the step that turns a consistent corpus romanization into something a user’s inconsistent romanization can hit.
4. Normalize with a small model call when the flag fires. One cheap call converting the query to native script, run only on flagged queries, is a small fraction of retrieval cost and is more robust than a table because it uses context to pick among candidates. Cache by normalized query string; the same handful of queries repeat constantly.
5. Log the flag. Once you can count romanized queries you can see what share of your traffic was silently failing, which is usually the number that gets the work prioritised.