What Code-Switching Is, and Why It Breaks Simple NLP Assumptions

Code-switching is a sentence that changes language partway through. Almost every component in a conventional text pipeline assumes that does not happen, and each one assumes it in a slightly different way — so when the assumption fails, it fails several times, in several places, with different symptoms.

The definition, and three things it is not

Code-switching is the alternation between two or more languages by one speaker within a single conversation, turn, or sentence. Linguists split it by where the switch happens. Inter-sentential switching puts the boundary between sentences: one sentence in Hindi, the next in English. Intra-sentential switching — often called code-mixing — puts the boundary inside a clause, and that is the case that breaks software, because there is no point in the text where you can cut and hand each half to a monolingual component.

Three neighbouring phenomena get confused with it, and keeping them apart is what stops you building the wrong fix:

• Borrowing is not switching. “Le weekend” in French or “der Computer” in German is a word that has entered the receiving language and taken its grammar. The speaker is not switching; the lexicon has moved. A borrowed word is correctly tagged as the host language.
• Transliteration is not switching. "yeh bahut accha hai" is Hindi written in the Latin alphabet. The language never changed; only the script did. A detector that keys on script will get this exactly backwards, which is the subject of detecting language in a mixed-script document.
• Translation is not switching. A bilingual product string that appears twice, once per language, is two monolingual texts in one file. Splitting it is a formatting problem with a clean answer.

The standard analytical frame is Carol Myers-Scotton’s Matrix Language Frame model, published in 1993, which holds that one language — the matrix — supplies the grammatical frame of the clause while the other, the embedded language, supplies inserted material. It matters here for one practical reason: the matrix language is often not the language that contributes the most words. That single fact is why word-count-based routing misfires, and it is worked through in why code-switched support messages get misrouted.

One sentence, labelled token by token

Here is a review of the sort that arrives in a real support inbox in India, labelled with the token-level inventory from the First Shared Task on Language Identification in Code-Switched Data (Solorio et al., EMNLP 2014) — the same label set the LinCE benchmark later reused, so the labels below are not invented for this page:

Yaar   ye     Flipkart  delivery  bahut  slow   thi    but    packaging
lang1  lang1  ne        lang2     lang1  lang2  lang1  lang2  lang2

theek  hai
lang1  lang1

lang1 = Hindi (romanised)   lang2 = English   ne = named entity

Two things in that row of labels are worth staring at. The first is that delivery, slow, but and packaging are English words sitting inside a Hindi grammatical frame: the verb thi is a Hindi feminine past copula agreeing with delivery, an English noun that has been assigned Hindi gender. The second is that the sentence is 10 tokens, of which 5 are English and 4 are Hindi. Any procedure that picks a language by counting is deciding a tie.

Which stage breaks first

Run that sentence through a conventional pipeline and the failures arrive in a fixed order, because each stage consumes the previous stage’s output.

1. Language identification. A document-level detector — fastText’s lid.176, Google’s CLD3, langid.py — returns exactly one label with a confidence. On the sentence above it returns something like __label__en at moderate confidence, or hi, depending on the model and on nothing you control. This is the first failure, and it is the important one: the API shape guarantees it. There is no value in the return type that means “both”.
2. Normalisation and stopword removal. Conditioned on the label from step 1. An English stopword list removes but and leaves ye, bahut and hai — high- frequency Hindi function words — as though they were content. A Hindi list does the reverse. Either way the surviving bag of words is skewed towards whichever language the detector did not pick.
3. Stemming or lemmatisation. Also conditioned on step 1. An English stemmer applied to thi or bahut produces garbage that is not a word in either language, and the garbage is stable, so it looks like a legitimate rare token downstream.
4. Tagging, parsing, sentiment, retrieval. Every model here was trained on monolingual text. It does not error; it produces a confident, wrong answer, which is worse.

Note what is not on that list. Tokenization by whitespace survives this sentence intact, because both languages are written in Latin script with spaces between words. The intuition that code-switching is a tokenizer problem is a reasonable guess and it is wrong for the most common pairs. It becomes right the moment one side of the switch is a script without word spacing, which is why Thai-English and Chinese-English behave differently from Hindi-English.

The assumption underneath all of them

Every stage above inherits one design decision: that a document has a language, singular, resolvable once, and that the answer is a useful key for selecting downstream components. That is not a bug in any one library. It is baked into the shape of the corpora — monolingual — into the evaluation sets, which are also monolingual, and into the function signatures, which return a string rather than a span-labelled sequence.

The fix is correspondingly structural rather than local. You do not repair a code-switched pipeline by improving the detector; a better detector still returns one label. You repair it by moving language from a document-level attribute to a token-level one, which is a different task with its own published training data and its own name — detecting the switch points inside a sentence.

Why this is not an edge case

The reason to care is that the affected populations are not marginal ones. Hindi-English mixing is the default register of urban Indian social media and customer messaging; Spanish-English mixing is ordinary in US-facing consumer products; Arabic dialects mixed with French or English are standard across North Africa. The LinCE benchmark, published at LREC 2020, exists precisely because these were being treated as noise: it covers Hindi-English, Spanish-English, Nepali-English and Modern Standard Arabic with Egyptian Arabic, across language identification, named entity recognition, part-of-speech tagging and sentiment.

The practical consequence is that a monolingual pipeline does not degrade evenly across your user base. It works for users who write in one language and fails for users who write in two — and the second group is defined by geography, so the failure lands on whole markets at once rather than on a random sample of requests. That is why this shows up in support metrics as a regional quality complaint long before anybody traces it to a language detector returning a string.