Chunking Text: Sizes, Overlaps and the Trade-offs

Every tutorial picks a number — 512 tokens, 1,000 characters, 50 of overlap — and none of them can tell you why. The number is downstream of a property of your documents, and once you know which property, the number is nearly obvious.

The wrong question

“What chunk size should I use” presumes the unit is length. The unit is answerability: a chunk is right-sized when it contains everything needed to answer the questions it will be retrieved for, and nothing that dilutes its embedding.

Both failure directions are real and they look different in production. Too small and the chunk is a fragment — a table row without its header, a clause without its definitions, a function body without the import that names its types. Retrieval succeeds and the answer is still wrong, which is the harder bug to see. Too large and the embedding is an average of several topics, so it ranks moderately for everything and strongly for nothing; retrieval quietly stops working for specific queries while looking fine on vague ones.

Split on structure, not on length

Documents already contain the boundaries you want. Use them first and let length be a cap rather than a rule:

What overlap is really for

Overlap is a patch for a boundary you were not able to choose well. If you split arbitrarily, some sentence spanning the cut is destroyed, and repeating the last 10–20% of each chunk at the start of the next makes it likely that one copy survives intact.

Which means overlap is a cost you pay in proportion to how bad your boundaries are. Split structurally and you can often take it to zero; split arbitrarily and you are paying 15% more storage, 15% more embedding cost, and — the part people forget — introducing near-duplicate chunks that compete with each other in the ranking and crowd out genuinely different results. If you use overlap, deduplicate before you assemble the prompt.

A related habit worth acquiring: read your chunks. Not a sample of embeddings, the actual text of thirty randomly selected chunks. It takes ten minutes and it surfaces problems no metric will — a splitter that cut every table in half, a PDF extractor that interleaved two columns, headers and footers repeated into every chunk, a navigation menu occupying a third of the corpus. Chunking bugs are usually obvious to a human and invisible to recall@k, because a systematically mangled corpus is mangled for the evaluation set too.

Giving a chunk its context back

The deepest problem with chunking is that a chunk is torn out of a document that gave it meaning. “The rate increased by 3% in the second quarter” is unretrievable: it names no company, no year, no rate.

Anthropic published an approach for this in September 2024 under the name Contextual Retrieval: before embedding, prepend to each chunk a short, model-generated sentence situating it in its parent document. They reported that contextual embeddings reduced the top-20 retrieval failure rate on their evaluation corpora by roughly a third, and that combining contextual embeddings with a contextual BM25 index cut it by about half. Those are their numbers on their corpora, and the method — not the number — is what transfers.

The cheap version of the same idea requires no model calls at all: prepend the document title and heading path to every chunk as literal text. It costs a few tokens per chunk and recovers most of the naming information that the split destroyed. The related pattern worth knowing is small-to-big: embed the small chunk so retrieval is precise, but expand to its parent section before putting it in the prompt, so the model reads something coherent.

The loop that gives you a number

Nobody can tell you your chunk size, but the experiment that produces it is small. It needs a labelled set, which is the only genuinely tedious part:

# 1. Fifty real questions, each labelled with the passage that answers it.
#    Write them from support tickets or search logs, not from imagination.
gold = [("How do I rotate an API key?", "docs/keys.md#rotation"), ...]

# 2. Sweep the configurations you are actually choosing between.
for size in (256, 512, 1024, 2048):
    for overlap in (0, 0.1, 0.2):
        index = build_index(corpus, size=size, overlap=overlap)
        hits = sum(any(src in r.source for r in index.search(q, k=5))
                   for q, src in gold)
        print(size, overlap, "recall@5 =", hits / len(gold))

# 3. Then, separately, grade end-to-end answers at the best two settings.
#    Recall@5 and answer quality do not always peak at the same place.

Step three is the one people skip and it is where the surprise lives: the configuration with the best retrieval recall is not reliably the one with the best answers, because a larger chunk that ranks slightly worse can carry the surrounding context the model needed. Measure the thing you actually ship.

One thing to build in from the start rather than retrofit: metadata on every chunk. Source path, document title, heading path, version and date, at minimum. It costs almost nothing to store, it is what makes citations possible, it is what lets you filter a query to a product version or a date range before the vector search runs, and it is what lets you re-index a single document without rebuilding everything. A chunk that is only an embedding and a string is a chunk you cannot operate.