Chunking Chinese Text for RAG Without Word Boundaries

Every off-the-shelf text splitter is built on the assumption that a space separates two words. Chinese writes 我今天要去北京开会 with no spaces at all, and the splitter does not fail loudly when it meets that. It falls through its separator list to the empty string and starts cutting on raw index arithmetic, which puts chunk boundaries in the middle of names, numbers and clauses. The fix is to give it boundaries that actually exist in the writing system.

Why the default splitter fails

The common recursive splitters carry a separator list that reads roughly: paragraph break, line break, space, then the empty string as a last resort. On an English paragraph the third entry does the work. On a Chinese paragraph the first two only fire at paragraph and line breaks, the space separator matches nothing inside a paragraph, and the splitter reaches the empty-string fallback — which means it slices the string at whatever index makes the size budget come out right.

That is not a rare edge case; it is the normal path for Chinese input. The visible symptom is chunks that begin mid-word: a chunk ending …成本控制的关 and the next beginning 键指标… has split 关键 (“key”) across two embeddings, and neither half means what the whole meant. Retrieval degrades quietly, because the index still builds, the vectors still have the right dimensionality, and nothing in the pipeline raises an error.

There is a second, quieter failure underneath it. Because the splitter measures in characters and the model bills and truncates in tokens, a budget that was tuned on English is not the budget you think it is once the text is Han characters. That arithmetic is worked out in setting CJK chunk size in tokens rather than characters, and this recipe assumes it.

What counts as a boundary in Chinese

Chinese has no orthographic word boundary, but it has a rich and very regular punctuation system, and that punctuation is what you segment on. The characters are fullwidth and distinct from their ASCII lookalikes, which is the detail that breaks naive regexes written by someone testing on English:

• Sentence-final — 。 (U+3002 ideographic full stop), ！ (U+FF01), ？ (U+FF1F), and ；(U+FF1B) when a document uses the semicolon as a hard stop. Note that none of these is the ASCII ., so a regex looking for a period matches nothing.
• Clause-level — ，(U+FF0C), 、(U+3001, the enumeration comma used between list items), ：(U+FF1A). These are your second-tier split points, used only when a single sentence exceeds the budget on its own.
• Paired and trailing — closing quotation marks 」』 ”, closing brackets ）》】, and the ellipsis ……. A sentence boundary that falls before a closing quote produces a chunk starting with a stray 」, so the split has to happen after any run of closing marks.

Real documents are also mixed. Technical Chinese contains Latin product names, ASCII digits, URLs and code, and the ASCII period at the end of an English sentence embedded in a Chinese paragraph is a genuine boundary. Include both alphabets in the terminator class.

The chunker

The shape is: split into sentences on the terminator class keeping the terminator attached, then greedily pack whole sentences into a chunk until the next one would exceed the token budget. A sentence longer than the budget on its own is re-split on the clause-level class, and only if that still fails does anything get cut by character index.

1. Install a tokenizer that matches the embedding model you will index with, so the budget is measured in the same units the model counts in.
2. Split on a zero-width lookbehind, so the terminator stays with the sentence it ends rather than starting the next chunk.
3. Pack greedily to the budget, tracking the running token count rather than recomputing it for the whole buffer each time.
4. Recurse on the clause class for any single unit that exceeds the budget, and hard-cut only as the final fallback.
5. Carry overlap in whole sentences, never in characters — see the overlap arithmetic for CJK documents.

import re
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
count = lambda s: len(enc.encode(s))

# Split *after* a terminator and after any run of closing marks that follows it.
SENT_END = re.compile(r'(?<=[。！？；!?;])(?=[^」』”）》】…]|$)')
CLAUSE_END = re.compile(r'(?<=[，、：,:])')

def split_units(text, pattern):
    return [u for u in pattern.split(text) if u.strip()]

def pack(units, budget, overlap_units=1):
    chunks, cur, cur_n = [], [], 0
    for u in units:
        n = count(u)
        if n > budget:
            # One sentence larger than the whole budget: split it harder.
            sub = split_units(u, CLAUSE_END)
            if len(sub) == 1:
                sub = [u[i:i + 200] for i in range(0, len(u), 200)]
            for s in pack(sub, budget, 0):
                chunks.append(s)
            continue
        if cur and cur_n + n > budget:
            chunks.append("".join(cur))
            cur = cur[-overlap_units:] if overlap_units else []
            cur_n = sum(count(x) for x in cur)
        cur.append(u)
        cur_n += n
    if cur:
        chunks.append("".join(cur))
    return chunks

def chunk_chinese(text, budget=320):
    return pack(split_units(text, SENT_END), budget)

The 200 in the last-resort branch is a character count, and it is the one place this chunker cuts blindly. It fires on things like an unpunctuated table row pasted into a paragraph. Leaving it in is deliberate: without it, a single pathological unit either overflows the model or throws.

Token arithmetic on a labelled sample

Take a labelled sample: a 4,000-character Chinese policy document, counted in Unicode code points, of which roughly 3,700 are Han characters and the rest punctuation and Latin. Call r the tokens-per-character ratio your tokenizer produces for this text — run count(doc) / len(doc) once and use the real number rather than a guess.

With a budget of 320 tokens, the number of chunks is approximately count(doc) / 320. If your measured r came out at 0.7 — used here purely as a stand-in so the arithmetic is followable, not as a claim about your tokenizer — the document is about 2,800 tokens and packs into nine chunks. If your tokenizer is an older vocabulary with thin CJK coverage and r comes out above 1.0, the same document is 4,000-plus tokens and needs thirteen. The document did not change; the chunk count moved by 40% on a number you have not measured.

The second number worth deriving is sentences per chunk. If the sample averages 40 characters per sentence, a 320-token budget at r = 0.7 holds roughly 457 characters, or about eleven sentences. That is a healthy chunk: enough context for a retrieval hit to be answerable, small enough that a single hit is not mostly irrelevant. If your arithmetic lands at two sentences per chunk, the budget is too small for the writing style and retrieval will return fragments.

Where this still breaks

• Documents with no punctuation. Subtitle files, chat logs and OCR output from low-quality scans often arrive with punctuation stripped. The sentence splitter returns one unit and every chunk goes through the blind fallback. Detect it: if the mean unit length after splitting is above a few hundred characters, the document has no usable boundaries and needs a word-segmentation library instead.
• Traditional and simplified in one index. 关键 and 關鍵 are the same word and different code points, so they embed differently and match differently under lexical search. Decide on one form and convert at index time, or index both.
• Vertical text and fullwidth spaces. Text extracted from vertically set material can carry U+3000 ideographic spaces that look like separators and are not reliable ones. Normalise them before splitting rather than adding them to the separator list.
• Headings absorbed into the following paragraph.Chinese headings frequently carry no terminating punctuation, so a sentence splitter glues a heading to the first sentence under it. Split on line breaks first and treat each line as a hard boundary, then apply the sentence splitter inside lines.