Chunk Overlap Strategy for CJK Documents

Overlap exists so that a fact straddling a chunk boundary survives in at least one chunk intact. Expressed as a percentage of a character count, it does that job at wildly different strengths depending on the script, because the number of propositions in 100 characters is not a constant.

What overlap is actually for

The failure overlap prevents is precise. A sentence says “the refund window is 30 days” and the previous sentence establishes that this applies to enterprise plans. Split between them and the chunk containing “30 days” does not contain “enterprise”, so a query about enterprise refunds does not retrieve it. Overlap fixes this by making the tail of each chunk also the head of the next, so at least one chunk holds both.

That framing already tells you the right unit. What has to survive is a sentence or two of surrounding context, not a number of characters. Characters are a proxy that happens to work in English because English sentence lengths in characters are fairly consistent.

It also tells you that overlap is not the only tool for the job, and often not the best one. The alternative is to give each chunk its context explicitly rather than by duplication: prepend the document title and the section heading to every chunk, or store a parent identifier and expand a retrieved chunk to its parent section before passing it to the generation model. Both attach the missing context without paying for it in duplicated vectors, and both are strictly better than overlap for the specific case of a chunk whose subject was established in a heading. Overlap remains necessary for the case a heading cannot fix: a fact whose qualifier is in the immediately preceding sentence.

The arithmetic, both ways

Work it with labelled assumptions rather than asserted averages, because sentence length varies by genre far more than it varies by language.

ASSUMPTION: English prose, ~5 characters per word (incl. the space)
ASSUMPTION: English sentences of ~18 words -> ~90 characters
ASSUMPTION: Chinese prose in your corpus averages S characters per
            sentence — measure it; a documentation corpus and a legal
            corpus will differ by a factor of two.

Case 1  English, chunk 1000 chars, overlap 10% = 100 chars
        100 / 90 ≈ 1.1 sentences carried forward.

Case 2  Chinese, chunk 1000 chars, overlap 10% = 100 chars
        with a STAND-IN S = 35 chars/sentence:
        100 / 35 ≈ 2.9 sentences carried forward.

Same setting, ~2.6× the semantic overlap.

Now the other direction — the one people actually hit, because the
chunk size gets fixed first (see the tokens-not-characters page):

Case 3  Chinese chunk sized in TOKENS to match the English budget of
        250 tokens, with a STAND-IN r_zh = 0.7 tok/char:
        250 / 0.7 ≈ 357 characters per chunk
        10% overlap = 36 chars ≈ 1.0 sentence at S = 35.

So a token-correct chunk size makes a percentage overlap come out
roughly right again — and a character-correct one does not.

The two cases pull in opposite directions and that is the reason this is confusing in practice. If you left the chunk size in characters, your CJK chunks are oversized and your overlap is oversized, and the two errors partly hide each other in the retrieval metrics while both inflate your index size and your context bill. If you fixed the chunk size to tokens and left overlap as a percentage, the overlap quietly became correct with it.

Why the two errors are not symmetric

Too little overlap and you lose facts at boundaries: a real retrieval failure that shows up as an answer of “the documents do not say” when they do. Too much overlap costs storage, embedding calls and duplicated hits in the top-k, and it is the last one that actually hurts. Three of your top eight chunks being near-copies of each other means the answer is built from five distinct passages rather than eight, and the model sees a repeated passage as corroboration it has not earned.

For CJK the duplication cost lands harder than in English, because a Han character carries more content per unit of overlap. Three overlapping chunks in Chinese can repeat an entire paragraph of argument, which is enough to visibly bias the generated answer toward whichever passage happened to be duplicated.

• Index size scales as 1 / (1 - overlap_fraction). A 10% overlap is 11% more vectors; a 50% overlap doubles them. That is embedding cost at ingest and storage cost forever.
• De-duplication is not free. Removing near-duplicate hits from a result set means a similarity comparison among the top-k, which is cheap in compute and awkward in ranking: dropping a duplicate promotes something from further down that may be worse than either copy.

Counting overlap in sentences

The robust formulation is to carry a fixed number of units, where a unit is whatever your segmenter produces — a sentence for Chinese and Japanese, a sentence or clause for Thai, a sentence for English. One or two units, regardless of language, regardless of chunk size.

# Overlap by units, not characters: the tail of chunk N is the head of N+1.
def pack_with_unit_overlap(units, budget, ntok, overlap_units=1):
    out, cur, n = [], [], 0
    for u in units:
        k = ntok(u)
        if cur and n + k > budget:
            out.append("".join(cur))
            cur = cur[-overlap_units:]
            n = sum(ntok(x) for x in cur)
        cur.append(u); n += k
    if cur:
        out.append("".join(cur))
    return out

This has three properties a percentage does not. It is stable when the chunk size changes. It is stable across languages, because the segmenter has already absorbed the language difference. And it never cuts inside a unit, so the overlap region is always a readable, embeddable piece of text rather than a fragment starting mid-clause — which matters because the overlap region is, by construction, duplicated into a chunk where it is the opening context.

The segmenters for the two hardest cases are on their own pages: punctuation-based segmentation for Chinese and sentence segmentation for Japanese.

Choosing the number

One unit of overlap is the default and it is right for most documentation. Move to two when the corpus is anaphora-heavy — text that says “this policy”, “the above”, “同上” — because the referent is a sentence or two back and a single carried sentence often is not enough to resolve it.

Go to zero overlap when the units are genuinely independent: FAQ entries, log lines, catalogue records, table rows. Overlap on independent records is pure duplication with no boundary to protect, and it is a common source of a retrieval index that is twice the size it needs to be.

There is one asymmetry specific to CJK worth building into the default. Chinese and Japanese both drop subjects and both rely heavily on topic continuity across sentences: a paragraph establishes its subject once and the following four sentences say nothing about who or what they are describing. An English paragraph repeats the noun or a pronoun far more often, so an English chunk starting mid-paragraph usually still names its subject and a Chinese one frequently does not. That is a reason to prefer two units of overlap in CJK where you would use one in English — not because the characters are denser, but because the anaphora reaches further back.

Whatever number you land on, validate it the same way each time: retrieve for a set of queries whose answers you know straddle a boundary in the current chunking, and check whether a chunk containing both halves now exists. That is a direct test of the thing overlap is for, and it is far more informative than a change in an aggregate retrieval score, which moves for a dozen reasons at once.

The one thing not to do is tune overlap before the chunk size is correct. Overlap is a fraction of a quantity, and if the quantity is measured in the wrong units, every conclusion you draw about the fraction is about the wrong thing.