Language Detection and Script Handling

Language detection is the classical NLP task with the widest gap between how easy it looks and how badly it fails. On a paragraph of clean prose it is essentially solved. On the inputs a real system actually receives — search queries, product titles, mixed-language chat — it is a coin toss dressed as an answer.

How it works, and why it is fast

Almost every language identifier is a character n-gram model. Count overlapping sequences of two to five characters, compare the profile against per-language profiles learned from training text, and return the best match. That is all. The reason it works so well is that character trigram distributions are enormously distinctive across languages — _th, the, he_ in English against _de, der, en_ in German — and the reason it is fast is that the whole computation is counting and a dot product, with no neural network anywhere.

This is the archetype for the cluster’s argument. Language ID runs in microseconds, on a CPU, from a model that fits in under a megabyte, offline, deterministically, for hundreds of languages. There is no version of “call a model to detect the language” that is better on any axis except the specific hard cases below — and on those, the honest answer is that the model is also unreliable, because the information is genuinely absent from the input.

The tools worth using

• fastText language identification. The lid.176 model covers 176 languages, and the compressed .ftz variant is under a megabyte. Returns ranked labels with probabilities, which is the property that makes thresholding possible. The usual default choice.
• CLD2 and CLD3. Google’s compact language detectors, extracted from Chrome. CLD2 in particular will report multiple languages with per-language byte percentages for mixed documents, which is the right output shape for web pages.
• langid.py. Lui and Baldwin’s tool (ACL 2012 system demonstrations), notable for being trained to be robust across domains rather than tuned to one, and for being a single self-contained Python file with the model embedded.

All three are pip-installable, all three run offline, and the differences between them matter far less than the handling of the cases below.

One property is worth insisting on when choosing: the tool must return a score, or ideally a ranked list of candidates with scores, rather than a bare label. Everything useful in the rest of this page depends on being able to say “the model is not sure”, and a library that returns only its top guess has removed the one piece of information you needed. This is a general point about small classifiers and not a quirk of language ID.

Seven edge cases that break pipelines

Short text. The dominant failure by volume. A three-word search query contains perhaps twenty characters — a dozen trigrams — which is not enough evidence to separate related languages. Accuracy degrades sharply below roughly twenty to thirty characters and the model does not warn you; it returns a label with a confident-looking score.

Code-switching. One message containing two languages is normal in large parts of the world and near-universal in technical chat. A single-label classifier must pick one, and the result depends on which language happened to supply more characters. If this matters, you need per-segment detection, not per-document.

Romanised text. Hindi, Arabic, Greek, Russian and Japanese are all routinely written in Latin script online. A script-aware detector loses its strongest signal, and models trained on native-script corpora frequently label romanised Hindi as English, Indonesian or Italian.

Mutually intelligible neighbours. Serbian, Croatian and Bosnian; Malay and Indonesian; Norwegian Bokmål and Danish; Czech and Slovak; Hindi and Urdu when transliterated. These pairs are close enough that even long samples are genuinely ambiguous, and the choice between them is often political rather than linguistic. If your downstream behaviour differs between two of these, merge them into one bucket rather than pretending the classifier can separate them.

Non-prose input. Code, URLs, JSON, product SKUs, addresses and numeric tables have no language. The detector will still return one, usually English or something surprising, with no signal that the question was meaningless. Filter these out by the ratio of letters to other characters before detecting.

Boilerplate contamination. A page of German content wrapped in an English navigation, cookie banner and footer may be majority English by character count. This is another reason boilerplate removal belongs before every other step in the pipeline.

Homoglyph and mixed-script text. Cyrillic а and Latin a are different codepoints that render identically, and text deliberately mixing them — spam evasion, or simply a bad copy-paste — produces character profiles matching nothing. Detecting the mix is itself a useful signal; see the confusables problem.

A threshold is not optional

The single change that fixes most of this: stop treating detection as a function returning a language, and treat it as a decision with an explicit “unknown” outcome.

import re, fasttext

model = fasttext.load_model("lid.176.ftz")
LETTERS = re.compile(r"[^\W\d_]", re.UNICODE)

def detect(text, min_chars=25, min_conf=0.65, min_letter_ratio=0.5):
    text = " ".join(text.split())
    if len(text) < min_chars:
        return None, 0.0, "too_short"
    letters = len(LETTERS.findall(text))
    if letters / max(len(text), 1) < min_letter_ratio:
        return None, 0.0, "not_prose"        # code, URLs, tables
    labels, probs = model.predict(text, k=2)
    lang = labels[0].replace("__label__", "")
    conf = float(probs[0])
    margin = conf - float(probs[1]) if len(probs) > 1 else conf
    if conf < min_conf or margin < 0.15:     # ambiguous neighbours
        return None, conf, "low_confidence"
    return lang, conf, "ok"

Three details are doing the work. The length check refuses to answer when there is not enough evidence. The letter-ratio check refuses to answer when the input is not prose. And the margin between the top two candidates catches the mutually-intelligible-neighbour case, which a confidence threshold alone misses because both candidates can score highly.

What the pipeline does with None is a product decision, and it should be an explicit one: fall back to the user’s interface locale, fall back to a multilingual model that does not need the language, or route to a human. All three are defensible. Silently treating unknown text as English is what produces a search index where a tenth of the corpus was stemmed with the wrong analyser and nobody can explain why recall is bad.