Embedding a Document That Mixes Two Scripts

Technical documentation written in Hindi is not written in Hindi. It is written in Hindi with the nouns in English, and the chunk you embed contains both scripts. The obvious move — split by script, embed separately — is usually wrong, and the reason it is wrong tells you what to do instead.

What a mixed-script document actually looks like

The mixture is not random and it is not evenly distributed. In a typical Hindi technical page you will find:

अपना Aadhaar कार्ड नंबर दर्ज करें और Submit बटन पर क्लिक करें।
API key को .env file में रखें — इसे कभी commit न करें।

Devanagari: grammar, verbs, function words, general nouns
Latin:      proper nouns, product names, UI labels, identifiers,
            file names, technical terms, code

The Latin material is concentrated in exactly the words that discriminate between documents. Product names, API names, file names and UI labels are the high-information tokens; the Devanagari supplies the sentence that connects them. That distribution is the single most important fact for deciding how to chunk, and it holds across Hindi, Arabic, Thai, Russian and Japanese technical writing alike.

A second shape appears in documents that are bilingual rather than code-switched: a Hindi section followed by its English translation, or a table with parallel columns. That is a different problem — two monolingual passages in one file — and it wants a chunking boundary at the section break, not script-level surgery. See chunking a bilingual document.

The script ratio decides where the chunk lands

Compute the script ratio of a chunk by counting characters per Unicode block: Devanagari is U+0900–U+097F, basic Latin is U+0000–U+007F. Then convert to tokens, because that is what the model pools over, and Devanagari tokenizes to far more tokens per character than Latin does.

A chunk that is 80% Devanagari by character may be 90% Devanagari by token, and the pooled vector sits accordingly — near the Hindi region of a space that separates by language before it separates by meaning. A chunk that is 30% Devanagari by character, which is common for a page that is mostly a code example with Hindi commentary, may land nearer the English region.

The practical consequence is that identical content in two chunks of the same document can be reachable from different query languages purely because of how much code happened to be in each chunk. That is worth knowing before you conclude your Hindi retrieval is broken: it may be working fine on the chunks that are actually Hindi and failing on the ones your chunker filled with a config file.

Why splitting by script makes it worse

The intuition behind splitting is reasonable — give each language its own clean vector — and it fails for three reasons.

• It severs the words from their meaning. Aadhaar separated from कार्ड नंबर is a bare proper noun with no indication of what the passage says about it. The Devanagari fragment, stripped of its nouns, is grammatical connective tissue: “enter your ... and click the ... button”. Neither piece retrieves well. Together they are a sentence.
• It produces short fragments. Splitting by script inside a sentence yields runs of two and three words. Short text embeds badly and unstably in every language, and it embeds worst in the morphologically rich one — the same effect described in why short queries embed worse in inflected languages.
• It throws away the script-invariant anchors. The Latin technical terms are the one part of the document that a query in any language can match, because they are spelled the same in the Hindi, Tamil and Portuguese versions of your docs. Keeping them inside the Devanagari chunk is what makes that chunk findable from an English query at all.

So the default is: do not split by script. Chunk on semantic boundaries — headings, paragraphs, list items — exactly as you would for a monolingual document, and let each chunk keep whatever mixture it has.

The case for a second vector

There is one situation where script does justify extra work, and it is not a split. It is an additional vector on the same chunk.

If a chunk is heavily dominated by one script but contains a meaningful minority-script passage — say a chunk that is 90% Latin code with three sentences of Devanagari explanation — the minority passage is effectively invisible. Its tokens are outvoted in the pooling, and no Hindi query will reach it. The repair is to keep the full chunk as the primary vector and add a second vector computed from the minority-script passage alone, retrieving with the maximum over a chunk’s vectors.

for chunk in chunks:
    vectors = [embed(chunk.text)]                  # always
    ratio = script_ratios(chunk.text)              # by unicode block, in tokens

    for script, share in ratio.items():
        # a real passage, but outvoted in the pooled vector
        if 0.05 < share < 0.35:
            passage = extract_runs(chunk.text, script, min_chars=80)
            if passage:
                vectors.append(embed(passage))

    index.add(chunk.id, vectors)   # retrieval scores max over vectors

The thresholds are the design decision. Below about five percent you are embedding a stray word or two and adding noise; above about a third the passage already influences the pooled vector enough to be found. The minimum length matters more than the ratio — a passage shorter than a sentence should never get its own vector, because a short fragment produces an unstable embedding that will match things it should not.

This costs storage proportional to how many chunks qualify, typically a small fraction, and it costs no recall on the chunks that do not. It is strictly a superset of the single-vector behaviour, which is why it is preferable to splitting: splitting trades one retrieval path for another, and this one adds a path.

Normalization traps specific to mixed script

Mixed-script documents concentrate several Unicode problems that monolingual text mostly avoids.

• Zero-width joiners survive NFC. Devanagari uses ZWNJ (U+200C) and ZWJ (U+200D) to control whether consonants form a conjunct ligature. Copy-paste from rendered text scatters them, and normalization does not remove them, so two visually identical strings differ in bytes and tokenize differently. Decide explicitly whether to strip them before indexing, and do the same to queries.
• Digits come in two families. Devanagari digits ०–९ and ASCII digits are different code points for the same numbers. A version number or an Aadhaar number can be written either way. NFKC folds them to ASCII, which is usually what you want for retrieval.
• Punctuation crosses scripts. The danda । ends a Devanagari sentence, and a mixed document uses it alongside the full stop. A sentence splitter that only knows about . will produce chunk boundaries in the wrong places, which is usually the actual cause of a “bad embeddings” report.
• Direction marks in Arabic-Latin mixtures. The same document shape with Arabic instead of Devanagari carries LRM, RLM and embedding controls from the authoring tool. Strip them for the index and keep them for display — see normalizing RTL text before indexing.