How Code-Switched Text Confuses Embedding Models

A support ticket written half in Hindi and half in English embeds to a single vector. That vector has to sit somewhere, and where it sits is decided by arithmetic over token representations — which means the language contributing more tokens wins, whether or not it carries the meaning.

A labelled sentence

Here is an ordinary sentence from a Delhi customer support queue, with each word labelled by language. This is intra-sentential code-switching: not two sentences in two languages, but one grammatical sentence drawing lexicon from both.

मैंने   कल    order  place  किया   लेकिन   delivery  अभी    तक    नहीं   आई
HI      HI    EN     EN     HI     HI      EN        HI     HI    HI     HI

"I placed an order yesterday but the delivery still hasn't arrived."

Words: 11 total — 8 Hindi, 3 English
Content words: order, place, delivery (EN) · आई/आना = arrive (HI)

Notice the structure, because it is typical and it is the reason this is hard. The grammar is Hindi: verb-final order, the ergative मैंने marker, the Hindi light verb किया doing the conjugation work for the English verb place. The domain vocabulary is English: order, place, delivery. This asymmetry is not accidental. In Hindi-English, Spanish-English and many other pairs, the matrix language supplies the frame and the embedded language supplies nouns and technical terms, because those are the words the speaker learned in that language.

Pooling weights by token, not by meaning

Almost every sentence embedding model produces its vector by mean pooling the final-layer token representations, or by taking a CLS position that attends over all of them. Either way, the sentence vector is a weighted combination in which each token gets a vote.

Now count tokens rather than words. English words in a model trained largely on English tokenize efficiently: order, place and delivery are likely one or two tokens each, say four tokens total. Devanagari is three bytes per character in UTF-8 and is a smaller share of the training mix, so the eight Hindi words may tokenize to twenty or more. Print the counts for your own model rather than trusting these — the ratio is the point, not the values.

Two things follow, and they pull in opposite directions:

• The Hindi half gets more votes. Twenty tokens against four means the pooled vector is dominated by the Hindi fragments, even though English carries the domain nouns.
• Each Hindi vote is worth less. Those twenty tokens are subword fragments that appear in a huge range of unrelated Hindi words, so their individual representations are diffuse. Many low-confidence votes plus a few sharp ones is a genuinely awkward combination: the sharp English tokens do not dominate the direction, and the diffuse Hindi ones do not point anywhere specific.

Add the language-identity component that multilingual spaces carry — a large, high-variance direction that encodes “this text is in Devanagari Hindi” before it encodes anything about orders or deliveries. A code-switched sentence has a mixed value on that direction, so it sits between the two language clusters: not squarely in the Hindi region where a Hindi query looks, and not in the English region where an English query looks.

Which half of the meaning survives

The prediction, and it is testable in ten minutes on your own index: the code-switched document is reachable from a query that mixes the same way, weakly reachable from a pure-English query that happens to use the same embedded nouns, and largely unreachable from a pure-Hindi query.

The last one is the counterintuitive part, since the sentence is mostly Hindi. A pure Hindi query for डिलीवरी (delivery, in Devanagari) does not match the document, because the document spells that concept delivery in Latin letters and the two share no tokens and no training association strong enough to bridge them. The Hindi in the document is grammatical scaffolding — अभी, तक, नहीं — which is exactly the material that carries the least retrieval signal. So the document is filed under the language of its function words while its content words are in the other language.

Practically, this is how a support queue misroutes. Tickets get clustered by which language supplied the grammar rather than by what they are about, and a topic that is always discussed with English nouns fragments across every matrix language in your user base.

Romanized code-switching is the harder case

The version above is the easy one, because the two scripts at least make the switch visible. The common form in the wild is romanized:

Maine kal order place kiya but delivery abhi tak nahi aayi
HI    HI  EN    EN    HI   EN  EN       HI   HI  HI   HI

Every token is now Latin script. The language-identity signal that at least separated the halves is gone, and the Hindi words are being tokenized by a merge table that has strong opinions about English letter sequences: kal, abhi and aayi get carved into fragments that carry English associations. There is no standardised romanization either, so nahi, nahin and nahi’n are three spellings of one word. Anything you build on top of a lookup table has to be many-to-many; see why transliterated queries fail.

Language identification also stops working here, and it fails silently by returning a confident wrong answer. Most detectors return a single label with a probability, and given romanized Hinglish they often return English with high confidence, because the script and several tokens genuinely are English. If a routing decision depends on that label, the decision is wrong and nothing in the logs says so. Research benchmarks for this exist — GLUECoS and LinCE, both published in 2020, collect code-switched tasks for Hindi-English and Spanish-English — but the detector in your pipeline was almost certainly not evaluated on them.

Indexing code-switched text so it can be found

• Index more than one vector per chunk. Store the original text as the primary vector and add a second vector for a normalized form — the text with the embedded-language terms mapped to their matrix-language equivalents, or the whole thing translated to one language. Retrieval takes the maximum over a chunk’s vectors. This costs storage and no recall.
• Keep the domain nouns in a lexical field. The English terms in a code-switched corpus are stable, spelled consistently and highly discriminative. BM25 over them is the most reliable retrieval path you have for this content.
• Expand queries across scripts. If a query arrives in Devanagari, also issue a romanized form; if it arrives in Latin, also issue a Devanagari transliteration. Fuse the result lists by rank.
• Do not route on a single language label. Treat language identification on short user text as a distribution, and route on a threshold with an explicit mixed bucket rather than on the argmax — see detecting language in code-switched text.