Handling Hinglish Text in an AI Pipeline

The hard part of Hinglish is not that two languages are present. It is that one of those languages arrives in two different scripts, often in the same conversation, and the two forms do not match each other under any string comparison you have.

What Hinglish actually looks like in a dataset

Pull a thousand rows of Indian consumer messaging and you will find at least four distinct forms in the same column:

1  "yeh product bahut accha hai"          Hindi, Latin script
2  "यह प्रोडक्ट बहुत अच्छा है"                Hindi, Devanagari, English loanword
3  "ye delivery bahut slow thi but ok"    Hindi matrix, English insertions
4  "The delivery thi slow, but chalta hai" English matrix, Hindi insertions

Rows 1 and 2 are the same sentence. No tokenizer, no embedding model and no exact-match index will treat them as related, because they share not one byte. Row 2 also contains प्रोडक्ट — the English word “product” written in Devanagari — which is an English borrowing transliterated into the Hindi script, the exact mirror of the problem in row 1. Rows 3 and 4 differ in matrix language, and that difference is invisible to word counts: row 3 has more English tokens than Hindi ones while being grammatically Hindi.

If your pipeline only ever sees one of these four, you have a sampling artefact and not a solved problem. Check the distribution before you design anything.

The script problem comes first

Deal with script before you deal with language, because script is deterministic and language is not. Every character has a Unicode script property; you can read it without a model and without a guess. Devanagari occupies U+0900–U+097F, and the presence of a single character in that block is proof, not evidence.

import regex  # not re — the standard library lacks \p{Script=...}

DEVA = regex.compile(r"\p{Script=Devanagari}")
LATN = regex.compile(r"\p{Script=Latin}")

def script_profile(text: str) -> dict:
    return {
        "deva": len(DEVA.findall(text)),
        "latn": len(LATN.findall(text)),
    }

script_profile("ye delivery bahut slow thi")   # deva 0,  latn 22
script_profile("यह delivery बहुत slow है")      # deva 8,  latn 13

That second profile — both counts non-zero — is the signal that you are holding a mixed-script document, and it should route differently from the first. It is cheap, it never mispredicts, and it runs before any model loads.

Romanised Hindi has no standard spelling

This is the property that makes Hinglish harder than Spanglish. Spanish written in Latin script has an orthography; romanised Hindi does not. The same word appears as:

accha  acha  achcha  achha  aacha       "good"
nahi   nahin  nhi   nahiin           "no / not"
kya    kyaa   kia                    "what"
hai    hain   hei   he               "is / are"

These are not typos to be corrected against a dictionary, because there is no authoritative dictionary of romanised Hindi to correct against. They are all valid. Any component that treats surface form as identity — an exact-match keyword rule, a stopword list, a bag-of-words feature, a lexicon-based sentiment scorer — fragments across these variants and each variant gets too little data to matter.

There are two workable responses and they suit different systems. If you control a lexicon-based component, map variants to a canonical form with a hand-built table for the few hundred high-frequency function words that actually carry the grammar — hai, nahi, kya, bahut, lekin, aur — and leave content words alone. If you are feeding a model, do not normalise at all: subword tokenizers already share prefixes across accha and acha, and multilingual encoders trained on web text have seen every one of these spellings. Normalising by hand in front of a model destroys information it was going to use.

The pipeline

1. Profile the script. Run the character-class count above. Record deva, latn and their ratio as fields on the record. Do not throw this away after routing — it is the most reliable feature you will have and it is free.
2. Transliterate Devanagari to Latin, not the other way. If you need one representation for indexing, go to Latin. Devanagari to Latin is lossy but well defined; Latin to Devanagari requires guessing vowels that romanised Hindi routinely omits, and the guess is wrong often enough to poison an index. This direction is worked through in Devanagari to Latin transliteration.
3. Detect language at token level, not document level. Use a token-labelling model rather than lid.176. Hindi-English is one of the four pairs covered by LinCE, so labelled training data for exactly this task exists publicly rather than needing to be invented.
4. Skip the monolingual preprocessing entirely. No stopword removal, no stemming, no spell correction. Every one of those is conditioned on a single language label you do not have, and each does more damage than the noise it removes.
5. Send the raw string to a multilingual model. Whatever you are doing downstream — classification, extraction, embedding — do it with a model whose training data included romanised Indic web text, and evaluate it on your own mixed sample rather than on a Hindi benchmark. A model that scores well on Devanagari Hindi can be poor on romanised Hindi; they are close to different languages as far as the tokenizer is concerned.

Where to stop normalising

The failure mode on this ground is over-processing. Each of the normalisation steps above is individually defensible and the stack of them is destructive: transliterate, then lowercase, then strip non-ASCII, then spell-correct against an English dictionary, and bahut accha becomes bahut acacia. That correction is stable and silent, and it will survive every test you have, because nothing downstream knows the string used to be Hindi.

The rule that holds: normalise for anything that compares strings, and do not normalise for anything that runs a model. An index, a deduplication key and a keyword rule all need canonical forms. A transformer needs the bytes the user typed. If one record has to serve both, keep two fields.

For the same problem in the Spanish-English pair, where the script question does not arise and morphology takes its place, see handling Spanglish text.