Setting a Confidence Threshold for Automatic Language Detection

Every language detector returns a number next to its label, and almost every integration picks a threshold for that number out of the air — 0.8, because it looks confident. The number is not a probability that the answer is right, and the correct cut-off for your traffic depends almost entirely on how long your strings are.

What the score is, and is not

The score is a softmax output over the detector’s label set. It is the model’s normalised preference among mutually exclusive single-language hypotheses, computed from pooled character n-grams. It is not calibrated: a 0.90 does not mean nine out of ten strings scoring 0.90 are correctly labelled, and there is no reason it should, because nothing in the training objective rewarded calibration.

Two things drag the score down, and they are not the same failure:

• Not enough evidence. Short strings, digits, emoji, URLs, product codes. The pooled vector is dominated by a handful of n-grams and lands somewhere between several labels.
• Genuinely ambiguous evidence. The string really is compatible with more than one language — a proper noun, a loanword, a code-switched sentence, or a pair of near-identical languages.

A single threshold treats both as “unknown”, which is usually the right operational answer but the wrong diagnostic one. The first is fixed by asking for more text; the second is not fixable at all.

Length is the variable that moves it

Label a handful of strings by hand and the pattern shows up immediately. These are all unambiguously English to a human reader:

"ok"                                        2 chars
"no"                                        2 chars   -- also Spanish, Italian, Polish, Norwegian
"thanks"                                    6 chars
"cannot log in"                            13 chars
"I cannot log in to the dashboard today"   38 chars

The first two carry essentially no discriminating n-grams. “no” is a complete, correctly spelled word in English, Spanish, Italian, Polish and Norwegian; there is no evidence in those two characters that favours one, so whichever label the model happens to weight highest wins by a margin that means nothing. “ok” is worse: it is not really in any language.

By around 30–40 characters of ordinary prose, a detector has enough n-grams that the score for the correct label separates clearly from the rest. Below roughly 15 characters, scores for the correct and incorrect labels overlap so heavily that no threshold separates them — you are choosing between accepting wrong answers and rejecting right ones with no setting that does both. This is why the most valuable line of code in a detection pipeline is often a length gate before the detector runs at all.

Deriving your threshold

Do this on your own traffic. A threshold derived from Wikipedia sentences is meaningless for chat messages, and a threshold derived from chat messages is meaningless for uploaded documents.

1. Sample 200–500 real strings from the actual field you will detect on, stratified by length: put at least fifty in each of the buckets under 10 characters, 10–30, 30–100, and over 100. Do not filter out the junk — the junk is what the threshold is for.
2. Label them by hand. Use the tag set you actually act on, plus two extra labels the detector cannot produce: mixed and none (for URLs, order numbers, emoji-only messages). You will need them.
3. Run the detector and record the top label with its score, plus the second label with its score. The gap between them is a better signal than the top score alone.
4. Sort by score and read the list. Do not compute an aggregate first. Walk down from the top and find the row where your hand labels start disagreeing with the detector. That row’s score is your empirical ceiling.
5. Pick the cut-off from a precision target, not from a round number. If acting on a wrong language is expensive (auto-translating a document, routing to a human queue), set the threshold where precision on your labelled set is above 95% and accept that a large share falls through to the fallback. If acting wrongly is cheap and reversible (choosing a UI default the user can change), a much lower cut-off is correct.
6. Re-run the whole procedure when you upgrade the detector. Score distributions are not stable across model releases, and a threshold copied forward silently changes behaviour.

import fasttext

model = fasttext.load_model("lid.176.bin")

MIN_CHARS = 20
THRESHOLD = 0.65     # derived from the labelled set, not chosen

def detect(text: str):
    cleaned = " ".join(text.split())
    if len(cleaned) < MIN_CHARS:
        return None, 0.0, "too_short"

    labels, scores = model.predict(cleaned, k=2)
    top, second = labels[0].removeprefix("__label__"), labels[1].removeprefix("__label__")
    gap = scores[0] - scores[1]

    if scores[0] < THRESHOLD:
        return None, float(scores[0]), "low_confidence"
    if gap < 0.20:
        return None, float(scores[0]), f"ambiguous:{top}/{second}"

    return top, float(scores[0]), "ok"

You need two thresholds, not one

The gap check in that function is the part most implementations are missing, and it is what separates the two failure modes from the first section. A string scoring 0.71 for hr with 0.24 for sr is a confident detection of a language pair the model cannot separate; a string scoring 0.71 for hr with 0.03 for everything else is a genuinely confident detection. The top score is identical. The correct action is not.

Where the two candidates are languages you treat identically — Bokmål and Nynorsk if you serve one Norwegian, Bosnian and Croatian and Serbian if you serve one Latin-script variant — collapse them into one label before applying the threshold. Summing the probabilities of labels you do not distinguish turns a rejected detection into an accepted one, correctly.

What to do below the line

A threshold is only useful if something happens underneath it. The full ladder is the subject of what to do when automatic language detection is wrong, but the short version is: never fall back to a hardcoded en. Fall back to a signal you already have — the account’s stored locale, the browser’s Accept-Language header, the language of the previous confident turn in the same conversation — and only then to asking.