Chunking for RAG: Size, Overlap and Semantic Splitting

Every chunking guide opens by recommending 512 tokens with 50 tokens of overlap. That number is a default, not an answer, and the reason it is unsatisfying is that chunk size is the wrong variable to be thinking about first.

A chunk is a retrieval unit, not a size

A chunk has to do two incompatible jobs. It must be findable: its embedding is a single point, so the narrower its topic the sharper that point is, which argues for small chunks. And it must be sufficient: whatever the model needs in order to answer has to be inside it, which argues for large ones.

Those two pressures do not resolve into a number. They resolve into a question about your documents: what is the smallest span of text that answers a question on its own? In an API reference that is one endpoint. In a legal contract it is one clause with its definitions. In a support transcript it might be a whole conversation, because the resolution is meaningless without the complaint. Find that unit and the token count follows from it, instead of the other way round.

The corollary is that a single global chunk size is only correct if your corpus is homogeneous. Mixed corpora — PDFs, wiki pages, chat logs, code — want different splitters per source type, and the machinery for that is one dispatch on MIME type at ingest.

The strategies, and what each one breaks

Parent–child is under-used relative to how well it addresses the actual problem. The index holds one embedding per sentence or per two-sentence window; the retrieval returns the enclosing section. You get a sharp match and a sufficient context, and the only cost is that your top-k now needs deduplication, because five sentences from the same section collapse to one parent.

What overlap costs

Overlap exists so that a fact straddling a boundary appears whole in at least one chunk. It is not free, and the arithmetic is exact. For a document of N tokens split at size s with overlap v, the stride is s - v and the chunk count is:

chunks = ceil((N - v) / (s - v))

N = 100,000   s = 512   v = 0    ->  196 chunks
N = 100,000   s = 512   v = 64   ->  224 chunks   (+14%)
N = 100,000   s = 512   v = 128  ->  261 chunks   (+33%)
N = 100,000   s = 512   v = 256  ->  391 chunks   (+99%)

That percentage is paid three times: in embedding calls at ingest, in index storage forever, and in every brute-force scan or graph traversal at query time. It also shows up as duplicate results, since two adjacent chunks sharing half their text will score similarly and both land in your top-k, spending a slot on text you already have.

A useful rule: overlap should be about the length of the longest sentence you cannot afford to cut, which for English prose is roughly 50 to 80 tokens. Beyond that you are buying duplication, not safety. And if you are using structural or semantic splitting, overlap is often unnecessary altogether — the boundaries are already in places where a fact does not straddle.

Six named failure modes

• The orphaned pronoun. A chunk begins “It does not apply to annual plans.” Nothing in it names the policy, so no query about the policy retrieves it. This is the single most common chunking defect and the reason contextual retrieval works.
• The header divorce. The section title “Refunds” ends up at the tail of the previous chunk and the refund text starts the next one, stripped of the only word that said what it was about. Fix: prepend the heading path to every chunk derived from that section.
• The split table. Rows 1–8 in one chunk, rows 9–20 plus the totals in the next, and the header row in neither. Tables need a splitter that repeats the header on each fragment or refuses to split at all.
• The boilerplate magnet. A page footer, cookie notice or licence header embedded a thousand times produces a thousand near-identical vectors that crowd the top-k for any vaguely legal query. Strip boilerplate at ingest; it is cheaper than fixing the symptom in ranking.
• Token-count drift. Splitting by characters and assuming four characters per token is fine for English prose and badly wrong for code, JSON, CJK text and anything with long identifiers. If a chunk must fit a limit, count tokens with the actual tokeniser.
• The invisible ingest bug. A PDF extractor that emits two-column text in reading order across the columns produces chunks that are grammatical nonsense, and the pipeline will not complain. Print twenty random chunks and read them before tuning anything. This finds more defects per minute than any other activity in this cluster.

A decision procedure

In order, stopping when one applies:

• Does the document have real structure — headings, sections, an enforced schema? Split on it, prepend the heading path, and stop.
• Is it code? Split on the syntax tree, not the text; a separate page in this cluster covers the splitter.
• Are there natural records — one ticket, one email, one product? The record is the chunk, however long or short it is.
• Otherwise: recursive splitting at 400–800 tokens with 50–80 of overlap, and revisit only when your evaluation set shows retrieval failures you can trace to a boundary.

That last clause is the operative one. Chunking is the stage where teams spend the most time tuning without a measurement loop, and it is the stage where the difference between a good and a mediocre choice is invisible until you have fifty questions with known answers to test against.