Why Sentiment Analysis Fails on Code-Switched Text

A code-switched review does not get a slightly worse sentiment score. It gets an inverted one, and the inversion has a precise cause: the polarity-bearing words and the operator that negates them are in different languages, and nothing in the pipeline applies one to the other.

The review that scores backwards

Here is a product review in Hinglish, with the tokens a sentiment component actually reacts to marked:

"Delivery   fast    thi    but    product   bilkul   bekaar   hai"
             ↑POS                    ↑INTENS  ↑NEG

 English polarity terms:  fast (+)
 Hindi polarity terms:    bekaar (-)   bilkul (intensifier: "completely")
 Overall human judgment:  strongly negative

An English-scoped scorer sees fast as positive, delivery and product as neutral, but as a contrast marker with no polarity of its own, and thi, bilkul, bekaar, hai as unknown tokens. Unknown tokens do not subtract; they contribute nothing. The score is positive. The review is a complaint.

The Spanish-English case produces the same result through explicit negation rather than through an unknown negative word:

"The food was amazing pero el servicio no valió la pena"
            ↑POS                        ↑NEG-OP

 English scorer:  amazing (+++), everything after "pero" unknown  →  positive
 Human judgment:  mixed, tilting negative

Why the negation does not apply

Negation handling in a sentiment component is a scoped rule: find a negation cue, then flip or dampen the polarity of terms within some window after it. The cue list is per-language. In an English configuration it holds not, no, never, n’t, without. It does not hold nahi, bilkul nahi, no as the Spanish adverb, or ma as the Arabic negator.

So the failure is not that the model does not understand the second language. It is that the operator and its operand are being handled by two different configurations, and the composition never happens. Three arrangements produce three different wrong answers:

• Positive word in the scored language, negation in the other. The negation is invisible and the score comes out positive at full strength. This is the worst case and the most common one, because English contributes the adjectives while the matrix language contributes the grammar.
• Negation in the scored language, polarity word in the other. The cue fires with nothing in scope. Some implementations then flip the nearest scored term instead, which can invert an unrelated clause.
• Both in the unscored language. The whole clause contributes zero, and the document score is decided entirely by the other half of the sentence — which is often the polite half.

The third case explains an effect that looks like a data problem rather than a modelling one: code-switched reviews cluster nearer neutral than monolingual reviews of the same product. Half of each sentence is being dropped, and dropping tokens pulls a score toward the middle.

Lexicon scorers fail differently from models

A lexicon-and-rules scorer — VADER and its descendants — fails as described above: loudly, deterministically, in a way you can reproduce by inspecting which tokens matched. That is the good case. You can find it, and the sign of the error is predictable from which language contributed the adjective.

A multilingual transformer fine-tuned on monolingual sentiment data fails more quietly. It has representations for both languages, so no token is literally unknown, and it will produce a confident score. But its fine-tuning set almost certainly contained no sentence that switched language mid-clause, so the composition of an English adjective with a Hindi negator is a structure it has never been supervised on. The output is plausible, well-calibrated-looking, and unvalidated on this input class. You cannot spot it by inspection; you can only spot it by evaluating on code-switched data, which is why the sentiment task in LinCE (LREC 2020) exists as a separate benchmark rather than as a subset of a Spanish one.

The intensity problem nobody mentions

Beyond polarity there is strength, and strength is where code-switching carries meaning that monolingual text does not. Bilingual speakers frequently switch language at the emotional peak of an utterance — the complaint arrives in the first language even when the rest of the message is in the second. In “Everything was fine but yaar bilkul bekaar service” the switch itself is a signal, and a pipeline that scores only English is discarding precisely the clause the writer cared most about.

This is a general property worth holding on to: in code-switched text the switch point is informative, not noise. Any design that normalises everything into one language before scoring destroys it. Translating the whole message to English first — the usual first instinct — flattens the register difference completely, and the trade-off in doing that is the subject of translating first versus prompting natively.

What to do instead

• Stop using per-language lexicons on mixed text. There is no configuration of a monolingual scorer that gets this right, because the composition crosses the configuration boundary.
• Evaluate on code-switched examples before you trust a number. Hold out a few hundred real mixed messages from your own traffic and label them. If your model is right on monolingual text and near chance on mixed text, your headline accuracy is an average over two populations and it describes neither.
• Prefer a generative model with the sentence intact. Asking an instruction-following model to judge sentiment on the raw code-switched string, with a rubric, avoids the scoped-negation problem entirely — there is no per-language cue list to miss. It is slower and costlier per item, and for mixed text that is usually the right trade.
• Report coverage next to accuracy. If 12% of your reviews are code-switched and you score them at chance, say so. A single accuracy number hides which users you are wrong about.

One further complication applies if you are doing aspect-based sentiment rather than document-level scoring. The aspect and the opinion about it routinely arrive in different languages — “el packaging estaba destrozado” pairs an English aspect term with a Spanish predicate. Aspect extraction keyed to a language-specific term list will find the aspect and miss the opinion, or the reverse, and the pair is what the task needs. Extract aspects from the raw string with a multilingual model rather than from a per-language gazetteer.

The same asymmetry affects retrieval over review corpora. Two reviews expressing the same complaint in different mixing ratios embed further apart than either does from an unrelated monolingual review, because language dominates the embedding geometry before topic does — the effect described in embeddings of code-switched text. If you cluster reviews to find themes, code-switched ones form their own cluster and the theme is never surfaced.