Telling Closely Related Languages Apart Automatically

Language detection is close to solved for English versus Japanese and nowhere near solved for Croatian versus Serbian, Indonesian versus Malay, Czech versus Slovak, or Bokmål versus Danish. The pairs a detector fails on are exactly the pairs somebody needs it to get right, and the reason is mechanical.

A labelled pair that looks identical

Two sentences with the same meaning, both in Latin script, labelled by hand:

hr : Moram raditi sutra, pa cu popiti kavu i otici na posao.
sr : Moram da radim sutra, pa cu popiti kafu i otici na posao.

shared, character for character: "Moram", "sutra", "pa", "cu",
"popiti", "i", "otici", "na", "posao"   -- 9 of 11 tokens

Two differences carry the entire distinction. The first is lexical: kavu against kafu (coffee), a single character in a single word. The second is syntactic: the Croatian sentence uses the infinitive raditi after a modal, where the Serbian sentence uses the da plus present construction da radim. Both patterns occur in both languages; they differ in frequency, not in grammaticality.

The third classic discriminator does not even appear here: the ijekavian and ekavian reflexes, mlijeko against mleko, vrijeme against vreme. That one is a reliable signal when the sentence happens to contain such a word and no signal at all when it does not — which is most sentences.

Why the n-grams cannot separate them

A detector represents each sentence as pooled character n-grams. Nine of eleven tokens above are byte-identical, so the overwhelming majority of the extracted n-grams are shared, and they are shared with equal weight because they genuinely occur with near-equal frequency in both languages. The pooled vectors for the two sentences sit almost on top of each other in the model’s feature space.

The discriminating evidence — one character inside kavu, and the presence of a two-letter function word da in a particular position — contributes a handful of n-grams out of roughly a hundred. Averaging is a low-pass filter: it preserves what is common and attenuates what is rare, which is the opposite of what this task needs. The signal is not absent from the input; it is destroyed by the pooling.

The same argument explains every pair on the list. Indonesian and Malay share most of their vocabulary and differ mainly in a set of specific words. Czech and Slovak differ in a set of diacritics and endings. Bokmål and Danish differ in orthographic conventions applied to a largely shared lexicon. In every case the distinguishing features are sparse, and sparse features do not survive averaging.

The score splits, and the split is honest

What you observe when you run such a sentence through a multi-label detector is the probability mass divided between the members of the pair, with everything else near zero. It is tempting to read that as the model being broken. It is not: it is the only accurate answer available from the evidence. Given a sentence whose observable features are consistent with both languages, a well-behaved classifiershould split.

The practical consequence is that a single confidence threshold rejects these sentences, and rejecting them is often wrong. If your product treats Croatian and Serbian in Latin script identically — same UI strings, same support queue, same model — then a 0.48 / 0.44 split is not a failure at all. Sum the probabilities of the labels you do not distinguish, and the merged label scores 0.92.

The features that do discriminate

If you genuinely need the distinction — for legal text, for regional pricing, for a translation vendor who bills per locale — build a second-stage binary classifier that runs only on strings the general detector marked as an ambiguous pair. It has one job, so it can use the sparse features directly instead of averaging them away.

• A closed lexical list. Twenty to fifty word pairs of the kava/kafa, tko/ko, tisuca/hiljada type. Presence of any member is strong evidence; absence is no evidence. Score as a sum of hits per side, not as a ratio.
• Morphological and syntactic patterns. The rate of modal-plus-infinitive against modal-plus-da. Counting occurrences across a paragraph is far more reliable than judging one sentence, which is the general lesson: for near-identical languages, the unit of decision should be the document, not the sentence.
• Script, where it exists. Serbian is written in both Cyrillic and Latin, so Cyrillic settles the question and Latin does not. This is the one case where the script property helps, and it helps in only one direction.
• Non-textual signals. Country from the request, currency, phone prefix, the stored account locale. These are outside the text and are frequently more accurate than anything inside it. Not using them because the problem was framed as text classification is a common and avoidable mistake.

The academic name for this task is discriminating between similar languages, and it has had its own shared tasks and its own workshop for years — the VarDial proceedings in the ACL Anthology are the place to look for which feature sets have held up on which pairs.

What to build

The design that survives contact with real traffic has three tiers. First, the general detector, used to narrow to a family. Second, a merge table that collapses the pairs your product does not care about, applied before any threshold. Third, a dedicated discriminator, run only on the pairs that survive and that you genuinely need split.

Everything else is a design decision about the product rather than a modelling problem. Deciding that you serve one Norwegian, or one Latin-script BCMS, or that you infer Simplified against Traditional Chinese from the character set rather than from a language label, removes the classification problem rather than solving it — and removing a problem you cannot solve reliably is the better engineering.