Setting CJK Chunk Size in Tokens, Not Characters

A chunk size of 1,000 is not a size. It is 1,000 of something, and almost every splitter defaults to counting characters while every model counts tokens. In English those two units happen to sit in a stable ratio, which is why the mismatch goes unnoticed until the corpus is Chinese, Japanese or Korean and the same setting produces chunks several times larger than intended.

The unit mismatch

Modern tokenizers are byte-level BPE: they merge frequent byte sequences into single tokens, and how many characters end up in one token depends entirely on how often those sequences appeared in the data the vocabulary was built from. English words are frequent, so common English words are one token each. A Han character is a three-byte UTF-8 sequence, and whether it costs one token or three depends on whether that particular character earned merges in the vocabulary. The per-language consequences of that are worked out in what a Chinese document costs in tokens; this page is about what it does to your chunk boundaries.

OpenAI publishes the English rule of thumb as roughly four characters per token for common English text, alongside tiktoken, its open-source tokenizer, which lets anyone check the number for any string. That figure is the anchor for everything below, and it is a rule of thumb the publisher describes as approximate, not a specification.

Measure the two ratios yourself

The derivation needs two numbers and both are one line of code. Run this against the encoding your embedding model actually uses — not a convenient one — because the whole point is that the ratio is vocabulary-specific.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")   # use YOUR model's encoding

def ratio(sample):
    return len(enc.encode(sample)) / len(sample)   # tokens per character

r_en = ratio(open("sample_en.txt").read())
r_zh = ratio(open("sample_zh.txt").read())
print(r_en, r_zh, r_zh / r_en)                     # the multiplier M

Three quantities come out of that: r_en, tokens per character for your English text; r_zh, the same for your CJK text; and the multiplier M = r_zh / r_en, which is the number that matters. Use a few thousand characters of representative prose, not a sentence — the ratio is unstable on short samples because a single rare character skews it.

The derivation

Assume you tuned a chunk size on English and it works: 1,000 characters, and the retrieval quality is what you want. Write the derivation with the assumptions labelled.

ASSUMPTION A: English runs at ~4 characters per token
                (OpenAI's published rule of thumb; r_en ≈ 0.25 tok/char)
ASSUMPTION B: your CJK ratio is r_zh, measured above.

Step 1  The English chunk you tuned:
          1000 chars × 0.25 tok/char = 250 tokens
          -> the real budget you validated is 250 TOKENS.

Step 2  The same setting applied to Chinese:
          1000 chars × r_zh tok/char = 1000·r_zh tokens

Step 3  The overshoot factor:
          (1000·r_zh) / 250 = 4·r_zh = M   (since r_en = 0.25)

Worked with a STAND-IN r_zh = 0.7 (substitute your measurement):
          Step 2  -> 700 tokens
          Step 3  -> 2.8× the validated budget

Worked with a STAND-IN r_zh = 0.55:
          Step 2  -> 550 tokens
          Step 3  -> 2.2×

So the answer to “how much does a character-count budget overshoot on CJK” is exactly 4 × r_zh for an English-tuned setting, and that lands between roughly two and three for the ratios typical of current tokenizers with reasonable CJK coverage. If your measured r_zh comes out above 1.0 — which happens with vocabularies that never learned merges for the characters in your corpus — the overshoot is four-fold or worse, and the same setting is producing chunks four times the size you validated.

The direction is worth stating explicitly because it is easy to get backwards. A Han character carries far more meaning than a Latin character, so 1,000 Han characters is much more content than 1,000 Latin characters — and it is also more tokens. Both go the same way. The chunk is bigger in every sense that matters.

What overshooting the budget costs

• Dilution in the embedding. A fixed-dimension vector summarises whatever it is given. A chunk holding three topics instead of one produces a vector that is near none of them, so the chunk ranks below a shorter, more focused chunk on every query it should have won.
• Silent truncation. Embedding endpoints have an input-token limit and several of them truncate rather than error. The tail of an oversized chunk is then indexed as if it were not there, and nothing in the response says so.
• Context blowout at answer time. Retrieving the top eight chunks at 2.8× the intended size is 22× the intended context budget, not 8×. This is where a pipeline that worked in English starts hitting context limits and cost alarms on a Chinese corpus.
• Overlap scaled wrong too. An overlap expressed as a character count inherits the same error, with a different practical effect — worked through in chunk overlap strategy for CJK documents.

Budgeting in tokens

One more thing the derivation does not capture: the three CJK languages are not one case. Japanese mixes kanji with two kana syllabaries and Latin in the same sentence, and kana generally tokenises worse than kanji because a kana string spells out a word the vocabulary may have merges for only in its kanji form. So r for Japanese moves with the script mix of the document, which varies between a technical manual and a customer-support transcript far more than it varies between two Chinese documents. Korean is different again: Hangul syllable blocks are three-byte sequences like Han characters, but Korean writes many more syllables per unit of meaning, so a Korean chunk holds fewer propositions per token than a Chinese one of the same size. Measure per language and, for Japanese, per document type.

The fix is not a per-language character table. It is to stop counting characters. Pass a length function that encodes with the right tokenizer and returns a token count; every mainstream splitter accepts one. The budget then means the same thing in every language, and the number you validated on English transfers without a conversion factor.

Two caveats. Tokenizing at ingest costs measurable CPU on a large corpus, so encode once per unit and cache the count rather than re-encoding the buffer at every packing step. And the embedding model’s tokenizer is often not the chat model’s; if you chunk with one and embed with another you have reintroduced the same mismatch in a subtler form.