Why Japanese Text Costs More Tokens Than English

Quoting a single token multiplier for Japanese hides the thing you can act on. Japanese mixes three scripts in one sentence, they cost different amounts, and the proportion of each varies enormously between a legal document and a product description.

One sentence, three writing systems

Here is a labelled example, broken down by script rather than treated as a block.

この機能はコンピューターのメモリを大量に使用することがあります。
"This feature can use a large amount of computer memory."

That is 32 code points. Six are kanji (機能, 大量, 使用), ten are katakana (コンピューター, メモリ), fifteen are hiragana, and one is the ideographic full stop. Every single one of them lives in U+3000–U+9FFF, so every one takes three UTF-8 bytes: the sentence is exactly 96 bytes. The English translation is 55 ASCII bytes. At the English rule of thumb of about four characters per token, the English is near 14 tokens.

Since byte-level BPE cannot exceed one token per byte, the Japanese sentence is bounded above at 96 tokens. Where it actually falls depends on which of the three scripts dominates, and they behave differently enough that averaging them is the mistake.

Kanji: dense, and mostly covered

Kanji behave much like Han characters in Chinese, and for the same reason: three bytes each, needing two merges to become one token, which common characters have earned and rare ones have not. The compensation is density. 機能 is two characters for what English spells with seven (“feature”); 使用 is two characters for “use” in its formal sense. Per character kanji look expensive; per unit of meaning they are the cheapest part of a Japanese sentence.

The failure case is proper nouns and rare readings. Personal and place names use kanji that are common as glyphs but rare in the specific two-character combination, and uncommon characters drop toward the byte floor of three tokens each. A page of names is a materially more expensive page than a page of prose, in a way no per-language multiplier predicts. The mechanism is the same one described in the arithmetic for Chinese, so it is not re-derived here.

Hiragana: cheap characters, long tails

Hiragana is a syllabary of fewer than a hundred distinct characters, all extremely frequent. That tiny, high-frequency inventory is exactly what BPE handles best: not only is each hiragana character almost certainly a single token, but frequent multi-character sequences get merged too. Grammatical strings like です, ます, という and ことが are common enough to plausibly occupy single merge slots.

The cost is not per character, it is volume. Japanese grammar is agglutinative in its verb endings, and politeness levels lengthen them: the plain 使う becomes 使用することがあります, where the meaning is carried by two kanji and the remaining nine hiragana are pure grammar. Formal Japanese therefore carries a long hiragana tail on every predicate, and every one of those characters is three bytes whether or not it is one token. Polite register genuinely costs more than plain register, and the difference is measurable on your own text with the script below.

Katakana: the expensive one

This is the part worth taking away. Katakana is used for foreign loanwords, and loanwords are precisely the words that are cheapest in English. コンピューター is seven katakana characters, 21 UTF-8 bytes, for a word that English writes as computer — eight ASCII bytes and, with a leading space, very likely a single token.

There is a second effect stacked on top, shared with Thai and derived in full there rather than here: Japanese has no spaces between words, so the tokenizer’s pre-tokenization step cannot cut the sentence into word-sized pieces and cannot attach a leading space to anchor a merge. A whole Japanese clause arrives at the merge stage as one long run. Japanese suffers this less than Thai does, because script changes between kanji, kana and katakana act as informal boundaries that the merge statistics can learn, but it is part of why even well-covered Japanese does not tokenise like a European language.

Worse, katakana loanwords are lexically open-ended. New product names, technical terms and brand names arrive in katakana constantly, so the merge table cannot have seen them: メモリ is common enough to be covered, but a katakana rendering of a new API name is not, and falls back toward two or three tokens per character. The long vowel mark ー (U+30FC) is its own three-byte code point, and it appears inside exactly the loanwords that are already expensive.

The derived consequence, with the assumption labelled: assume covered katakana runs at roughly one token per character and uncovered ones at two. A technical Japanese document that is 30% katakana by character costs meaningfully more than a literary one that is 5% katakana, even at identical character counts. That is a ratio you can steer — for terms that have a standard kanji form or an accepted English spelling, the choice of orthography is a cost decision as well as a style one.

Measuring the split in your own text

Because the three scripts differ, the useful measurement is per script, not per document. This counts tokens for each contiguous run and prints the cost by writing system.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def script_of(ch):
    o = ord(ch)
    if 0x3040 <= o <= 0x309F:
        return "hiragana"
    if 0x30A0 <= o <= 0x30FF:
        return "katakana"
    if 0x4E00 <= o <= 0x9FFF:
        return "kanji"
    return "other"

s = "この機能はコンピューターのメモリを大量に使用することがあります。"

runs = []
for ch in s:
    k = script_of(ch)
    if runs and runs[-1][0] == k:
        runs[-1][1] += ch
    else:
        runs.append([k, ch])

for kind, text in runs:
    print(kind, text, "chars", len(text), "tokens", len(enc.encode(text)))

Tokenising runs separately gives slightly different totals than tokenising the whole string, because merges can cross a script boundary — the total is the number to trust, and the per-run figures are for attribution. What you are looking for is which script is consuming your budget. If katakana is 15% of your characters and 30% of your tokens, that is where the money is going, and it is the one component you can influence without changing what the document says.

• Sentence splitting is a separate problem. Japanese has no spaces, so token budgets and chunk boundaries have to be computed independently — see segmenting Japanese for retrieval.
• Fullwidth and halfwidth digits are different tokens. １２３ (U+FF11–13) and 123 are unrelated byte sequences, and the fullwidth forms tokenise far worse.
• Romaji is cheap and lossy. Writing Japanese in Latin script cuts the token count sharply, but changes what the model sees and degrades quality; it is a compression trick with a real cost.