How Many Languages Models Really Support: Claimed Against Evaluated

A model card claiming support for a hundred languages and a benchmark evaluating twenty are not in conflict; they are measuring different things. This page separates the claims from the evaluations, names the benchmarks, and derives the cost penalty that applies even where support is genuine.

Four things “supports” can mean

A vendor listing supported languages is normally claiming the first, sometimes the second, and almost never the fourth. That is not dishonest — it is what the list is for — but it means the list cannot answer “can I deploy this to users in that language”.

The denominator: how many languages exist

SIL International’s Ethnologue, published annually, catalogues roughly seven thousand living languages, and Glottolog maintains an independent catalogue of comparable scale with different inclusion criteria. That is the denominator, and it makes even the most ambitious multilingual claims small.

  coverage  =  languages_supported  /  living_languages

  at 7,000 living languages:

    a model claiming  20 languages   ->  0.29%
    a model claiming 100 languages   ->  1.43%
    a model claiming 200 languages   ->  2.86%

Weighting by speakers rather than by language changes the picture
entirely, because language sizes are extremely unequal: a few dozen
languages account for the large majority of speakers. Both framings
are legitimate and they support opposite headlines, so state which
you are using.

Both denominators are worth carrying. Speaker-weighted coverage is what matters for a consumer product; language-count coverage is what matters for whether the technology reaches languages that already lack digital infrastructure.

The benchmarks that actually evaluate it

These are the named, public multilingual evaluations. Each states its language list, which is the property that makes it useful.

• FLORES-200 (Meta, released with the No Language Left Behind work in 2022). Parallel sentences across 200 languages, professionally translated from the same source content. Because it is genuinely parallel, it supports like-for-like comparison across languages in a way that independently-authored test sets cannot.
• Belebele (Meta, 2023). Multiple-choice reading comprehension over the FLORES passages, covering 122 language variants. One of the few benchmarks that tests comprehension rather than translation across a wide language set.
• MGSM (Shi et al., 2022). Grade-school mathematics problems translated into a small set of languages, designed to test whether reasoning transfers. The small language count is deliberate: it is a depth benchmark, not a coverage one.
• Translated MMLU variants. MMLU rendered into other languages, which tests knowledge and reasoning at the cost of inheriting every problem with MMLU itself, plus translation artefacts in questions that were culturally specific to begin with.
• Aya (Cohere For AI, 2024). A large open instruction-tuning dataset and evaluation suite spanning around a hundred languages, produced substantially through a community annotation effort. Notable because the data is open, so the language coverage can be inspected rather than trusted.
• XTREME, XCOPA, TyDi QA. Earlier cross-lingual benchmark suites, still useful for tasks that the newer suites do not cover, particularly typologically diverse question answering.

The tokenizer tax, derived

Even where support is real, a language can cost several times more than English to use, because tokenizers are trained predominantly on English text and split other scripts into more pieces. This is fully derivable and its consequences are larger than most people expect.

Let r be the ratio of tokens to English tokens for the same content.

Three separate penalties follow, all with the same factor:

  1. Cost.            price is per token, so cost multiplies by r.
  2. Context.         a fixed window holds 1/r as much content.
  3. Speed.           generation is per token, so the same amount of
                      output takes r times as long.

Worked at r = 2.5, which is in the range commonly observed for scripts
poorly served by a Latin-trained vocabulary:

  cost:     a document that costs $1.00 in English costs $2.50
  context:  a 128,000-token window holds
              English:  128,000 / 1.30 = 98,500 words
              at r=2.5: 128,000 / 3.25 = 39,400 words
  speed:    an answer that streams in 10 seconds takes 25

Combined, a user in that language pays 2.5x more, waits 2.5x longer,
and can supply 2.5x less context. None of this appears in any
"supported languages" list.

Measure the ratio for your own language and tokenizer rather than taking one from an article; it differs substantially between tokenizers and it has improved with each generation of larger vocabularies.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

pairs = [
    ("The quick brown fox jumps over the lazy dog.",
     "Le rapide renard brun saute par-dessus le chien paresseux."),
]

for english, other in pairs:
    e = len(enc.encode(english))
    o = len(enc.encode(other))
    print(f"english {e:4d}  other {o:4d}  ratio {o / e:.2f}")

# Use genuinely equivalent content — a professional translation of the
# same paragraph, not a machine translation and not a different text.
# FLORES-200 exists precisely to provide such pairs, across 200
# languages, which makes it the right corpus for this measurement.

There is a fuller treatment of the mechanism in the tokenizer language tax and of the tokenizer differences themselves in comparing tokenizers.

Where the asymmetry comes from

The capability gap is downstream of a data gap, and the data gap is measurable. Common Crawl publishes the language distribution of each crawl, and it is heavily skewed: a handful of languages account for the large majority of pages, and hundreds of languages appear in quantities too small to train anything on.

• The web is not proportional to speakers. Languages with many speakers and little web presence are the ones where models underperform most sharply relative to how many people they would serve.
• Language identification is itself unreliable for short texts, closely related languages and code-switched text, so the published distributions understate some languages by misclassifying them. See language detection.
• Much of the low-resource web is machine translation. Filtering pipelines increasingly try to detect and remove it, because training on machine-translated text of a language teaches the model the translator’s errors.
• Scripts and orthographies vary within a language. A language written in two scripts, or with unstandardised orthography, splits its already-small data further.
• Instruction data is far scarcer than raw text. A model can have plenty of pretraining text in a language and almost no instruction-following data, which produces exactly the failure of fluent-but-disobedient output described in the first section. This is what the Aya project was built to address.

Testing a language yourself

1. Take twenty to fifty real inputs in the target language from your own domain. Not translations of English test cases — real inputs, because the register and the subject matter differ.
2. Have a fluent speaker rate the outputs. There is no substitute and no automated metric that is trustworthy here; automated translation metrics measure similarity to a reference, not usefulness.
3. Test instruction adherence separately from fluency. Ask for JSON, for a fixed length, for a specific format, and check whether compliance drops relative to the same prompt in English. It usually does, and by more than fluency does.
4. Test refusals and safety behaviour explicitly in the target language. Do not assume it transfers; the published evidence is that it transfers less well than capability, and it is the failure with the worst consequences.
5. Measure the token ratio on your own content with the script above, and put the resulting cost and latency multipliers into your estimates before committing to a launch.
6. Compare at least two models. Coverage differs sharply between model families and correlates poorly with headline capability — the best model in English is often not the best in a given other language, and a smaller model trained with more of that language can beat it outright.