Handling Spanglish Text in an AI Pipeline

Spanish and English share an alphabet, so the script problems that dominate Hinglish are absent. What replaces them is morphology: Spanglish generates real, productive words that appear in neither language’s lexicon, and no amount of spell correction will map them back.

What Spanglish is in a dataset

Take a support message from a US consumer product with a bilingual user base, labelled with the same token-level inventory used across this cluster:

No     manches   the   app   se     crasheó  otra   vez   when
lang1  lang1     lang2 lang2 lang1  mixed    lang1  lang1 lang2

I      tried  to     upload  mis    fotos
lang2  lang2  lang2  lang2   lang1  lang1

lang1 = Spanish   lang2 = English   mixed = morphologically mixed token

Fifteen tokens: 7 Spanish, 7 English, 1 that is genuinely both. The switch happens twice, mid-clause, and both switch points fall at a syntactic boundary rather than at random — which is the usual finding, and the reason switch points are predictable enough to model at all.

One structural note that separates this from the Hindi case: the determiner-noun sequence mis fotos stayed entirely Spanish, while the app stayed entirely English. Switching inside a determiner phrase is rare in natural speech, and that regularity is useful — a model that predicts a switch in the middle of the plus noun is almost certainly wrong.

The words that are in neither dictionary

crasheó is an English verb stem, crash, carrying the Spanish third-person singular preterite ending -eó. It is not a typo, not slang in the disposable sense, and not a borrowing that has settled — it is generated on demand by a productive rule, which means the set of such words is open and cannot be enumerated in advance:

textear     text  + -ear      "to text"        →  texteé, textea, texteando
parkear     park  + -ear      "to park"        →  parkeé, parquea
googlear    google + -ear     "to google"      →  googleé, googleando
likear      like  + -ear      "to like"        →  likeé, likea
formatear   format + -ear     "to format"      →  formateé
lonchear    lunch + -ear      "to have lunch"  →  lonchée

Each of these inflects across the full Spanish verb paradigm, so one borrowed stem produces dozens of surface forms. A Spanish lemmatiser fails on all of them because the stem is not Spanish; an English lemmatiser fails because the suffix is not English. A spell checker in either language proposes a nearby real word — parkear becomes parear, likear becomes licuar — and the correction changes the meaning entirely while producing output that passes every validity check downstream.

The same rule runs on nouns and adjectives (la troca, el bil, rufo) and on English words taking Spanish gender and number (los emails, una app, los usuarios premium). The productive-morphology case is the one worth engineering for, because it is the one that grows.

Where the classifier misroutes it

Feed the example sentence to a document-level detector and you get one label from a near-tie. The consequence is not that the label is wrong — it is that it is unstable. Add three English words to the message and it flips. That instability is what makes the downstream symptom so confusing: two messages from the same user, about the same issue, land in different queues, and neither the routing rule nor the detector has a bug in it.

The specific misroute is worth naming because it is the one people actually hit. Spanish-language support queues are usually staffed separately, so the routing rule is literally if lang == "es". A bilingual user writing mostly in English with Spanish discourse markers gets routed to the Spanish queue and receives a reply in formal Spanish they did not ask for; a bilingual user writing mostly in Spanish with English technical nouns — app, upload, password, refund — gets routed to English, which is the worse direction, because technical nouns are exactly the ones English contributes. The full failure and the fix are in why code-switched support messages get misrouted.

The pipeline

1. Normalise Unicode, and only Unicode. Apply NFC so that é as a single code point and e plus a combining acute compare equal. This is the one normalisation that is always safe; see the difference between NFC and NFKC for why the more aggressive form is not.
2. Do not strip accents. It is tempting because it makes English and Spanish forms collide, and it is wrong: papá and papa, año and ano are different words, and one of each pair is embarrassing in a generated reply.
3. Label language per token, then aggregate deliberately. Spanish-English is the best-resourced code-switched pair in the public benchmarks — it appears in LinCE for language identification, named entities, part-of-speech and sentiment — so a token-level model here is a fine-tune rather than a research project.
4. Decide the matrix language from function words, not counts. Determiners, auxiliaries, pronouns and prepositions come from the matrix language; content nouns are the ones that get borrowed. Counting only closed-class tokens gives a far more stable answer than counting all tokens, and it is a dozen lines of code.
5. Route on the matrix language, reply in the user’s choice. The matrix language is your best guess at what the user thinks in. It is still a guess, so let the account setting override it, and let the reply language be a stored preference rather than a per-message inference.

The accent problem you inherit for free

Spanglish text is frequently typed on a keyboard configured for English, so accents and ñ are dropped by the writer, not by your pipeline: anos for años, si for sí, esta for está. You now have unaccented Spanish that is not a normalisation artefact and cannot be fixed by normalisation, because the information was never typed.

This is why accent-insensitive matching belongs in your search layer rather than in your storage layer. Fold accents at query time so anos retrieves años, and keep the original bytes in the record so a generated reply spells the user’s name correctly. Doing it the other way — folding on write — is unrecoverable, and names are where it hurts.