What It Costs to Build a Tokenizer for a New Language

People ask this expecting an answer in GPU-hours. Training a tokenizer is CPU work that finishes over lunch. The cost is entirely in getting enough clean text in the language to train it on, and in the vocabulary slots you have to take away from something else.

The compute is hours of CPU, not GPU-months

A subword tokenizer is not a neural network. Byte-pair encoding, the algorithm behind most current tokenizers, is a counting procedure: start from characters or bytes, repeatedly find the most frequent adjacent pair in the corpus, merge it, and record the merge. Unigram language model tokenization, the other common choice, starts from a large candidate vocabulary and prunes it by expectation-maximisation. Both are single-machine CPU algorithms with no gradient anywhere.

SentencePiece — introduced by Kudo and Richardson at EMNLP 2018 and still the standard implementation for both algorithms — trains a vocabulary on a few gigabytes of text in a matter of hours on a normal machine. The paper is arXiv 1808.06226. The Hugging Face tokenizers library, written in Rust, is faster still for BPE. Neither needs a GPU, and neither is a meaningful line item next to anything else in a model project.

The one resource that does bind is memory. SentencePiece loads training sentences into memory, which is why it exposes an input_sentence_size parameter that samples the corpus rather than reading all of it, and a shuffle_input_sentence flag to make that sample representative. Hitting that limit is the normal experience on a large corpus, and the sampling is a legitimate answer rather than a compromise: merge frequencies converge well before you have read everything.

import sentencepiece as spm

spm.SentencePieceTrainer.train(
    input="corpus.txt",
    model_prefix="mylang",
    vocab_size=32000,
    model_type="unigram",         # or "bpe"
    character_coverage=0.9995,    # see below — this default matters
    input_sentence_size=2_000_000,
    shuffle_input_sentence=True,
    byte_fallback=True,           # unseen characters become byte tokens
)

The corpus is the cost

The question “how much text do I need” has no single published minimum, and anyone quoting one precise number should be treated with suspicion. What the practice of the field indicates is a range with clear consequences at each end:

• Tens of megabytes produces a vocabulary that is technically valid and statistically thin. The merges reflect the quirks of whatever you had — if half your corpus is a Bible translation, religious vocabulary gets whole-word tokens and everyday speech gets fragmented. This is the realistic starting position for most languages that lack a tokenizer, and the resulting skew is the main quality problem.
• Hundreds of megabytes to a few gigabytes is where frequency estimates for a 32k vocabulary become stable and domain-balancing becomes possible. This is roughly the scale SentencePiece was designed around and it is a reasonable target.
• Beyond that, returns fall off sharply. A tokenizer learns a frequency ranking, and rankings converge long before language models stop improving. Doubling from 10GB to 20GB changes very little.

For a language with no substantial digital corpus, that acquisition is the entire project: sourcing text, clearing rights, removing machine translation, and — critically — checking that the corpus is not dominated by one genre. Machine-translated content is the specific contaminant to watch for, because it is abundant, it looks clean, and it teaches the tokenizer the segmentation of the source language rather than the target. The practical shape of that work is covered in building for a language with no digital corpus.

The vocabulary budget is a zero-sum decision

The cost that actually constrains you is not money or time. It is that vocabulary size is fixed at model design time, and every slot given to a new language is taken from an existing one.

Published vocabulary sizes show the trend clearly: GPT-2 used 50,257 tokens, Llama 2 used a 32,000-token SentencePiece vocabulary, Llama 3 moved to 128,256, and Gemma shipped 256,128. The growth is largely multilingual coverage. But vocabulary size is not free — the embedding matrix and the output projection both scale with it, so a large vocabulary costs parameters and memory in every forward pass, and the output softmax gets more expensive.

So the honest framing of “adding a language” to an existing tokenizer is: you need on the order of thousands of tokens for a language to be encoded efficiently rather than falling back to bytes, and they come from somewhere. Extending a vocabulary after the fact is possible — you append tokens and resize the embedding matrix — but the new embeddings are untrained, so the model must be further trained to use them. That step is real GPU work, and it is where the cost that people were originally asking about actually lives.

Character coverage, and the parameter that bites

character_coverage is the parameter that quietly decides whether your tokenizer works at all. It sets the fraction of characters in the training corpus that get their own symbol; the rest fall back to bytes or to an unknown token. SentencePiece’s documented guidance is to use 0.9995 for character-rich scripts such as Chinese and Japanese, and 1.0 for scripts with small character inventories.

For a new language the decision turns on the script rather than the language. An alphabet or abugida with a few dozen to a few hundred characters should use 1.0 — there is no reason to drop any of them. A script with a large inventory needs less than 1.0 or the rare characters consume vocabulary slots that would be better spent on frequent sequences. Getting this wrong in the permissive direction gives you a tokenizer that silently emits unknown tokens for rare but entirely valid characters, and the failure surfaces much later as text that round-trips incorrectly.

Setting byte_fallback=True is the safety net: unseen characters are encoded as their UTF-8 bytes rather than lost. It costs several tokens per unseen character, which is expensive, but expensive is better than unrecoverable — and it means combining marks, rare diacritics and emoji survive the round trip.

When a new tokenizer is not the answer

Before building one, check whether the problem is really segmentation. Two cheaper resolutions come first.

If an existing multilingual tokenizer already encodes the language at a tolerable rate, the bottleneck is model quality rather than tokenization, and a new tokenizer will not touch it. Measure before assuming: encode a paragraph with the candidate tokenizer and divide by the word count. A ratio near 1.5 tokens per word is normal; a ratio above 4 or 5 means the language is being spelled out and there is real headroom.

If the language shares a script with a well-supported one, byte-level fallback may already be handling it adequately, and the deficit is in training data rather than in the vocabulary. The distinction — and how to tell which one you are looking at — is the subject of the tokenizer vocabulary bottleneck in low-resource languages.