Text Chunker
Split your own text four different ways and read the actual chunks, with the overlap arithmetic shown.
Mean ~87 estimated tokens each, largest ~92.
- Document
- 980 chars · ~225 tokens (est.)
- Measured density
- 4.36 chars per token
- Chunk size used
- 400 chars
- Overlap used
- 80 chars
- Stride (size − overlap)
- 320 chars
- ceil((N − overlap) ÷ stride)
- 3 chunks
- Chunks produced
- 3
- Characters emitted
- 1,140 (1.16× the document)
- Tokens you will embed
- ~262 (est.)
usage object on an actual API response: that is what you are billed on, and it includes chat-template wrapping this model cannot see. Chunk sizes are enforced in characters; when you pick “estimated tokens” the size is converted using this text’s own measured density (4.36 chars per token), not a fixed constant. The chunk count for the two fixed strategies is ceil((N − overlap) ÷ (size − overlap)), which is printed above so you can check it against the list — dividing N by the size is the classic off-by-several error, and it undercounts every time you use overlap. Sentence and paragraph packing are greedy and never exceed the size; a single sentence longer than the size is hard-split rather than dropped, and paragraph mode falls back to sentences inside any block that is too big.All 3 chunks · the repeated head of each chunk is shaded
Retrieval quality is decided long before the query arrives. It is decided when the document is cut up, because a chunk is the smallest thing the retriever can return, and a retriever cannot return half of one. Cut too small and each chunk loses the sentence that gave it meaning. A paragraph about refund windows, split at 200 characters, produces a chunk that says "within this period" and never sa
at 200 characters, produces a chunk that says "within this period" and never says which period. The embedding is fine. The answer is wrong.
Cut too large and the opposite happens. One chunk now covers refunds, shipping and warranty; its embedding is the average of three topics and close to none of them. It ranks below a smaller chunk that is about exactly one thing.
Overlap is the usual compromw a smaller chunk that is about exactly one thing.
Overlap is the usual compromise. Repeating the tail of each chunk at the head of the next means a sentence that straddles a boundary appears whole in at least one chunk. You pay for the repetition twice: once when you embed it, once in every context window that retrieves both neighbours.What the chunk count actually depends on
The number people expect is the document length divided by the chunk size. That is right only when the overlap is zero. With overlap, each chunk after the first starts overlap characters back from where the previous one ended, so the window advances by a stride of size − overlap, not by size. The count is ceil((N − overlap) ÷ (size − overlap)). At a size of 400 with an overlap of 80 the stride is 320, and a 10,000-character document produces 32 chunks rather than the 25 the naive division predicts — a 28% larger embedding bill than budgeted for.
That multiplier is the line worth watching above. Overlap duplicates text, so “characters emitted” exceeds the document, and you pay for the excess twice: once at embedding time, and again in the context window every time retrieval returns two neighbouring chunks that share a tail.
The two structural strategies trade that predictability for coherence. Sentence packing never cuts mid-sentence, so no chunk ends on “within this”, but chunk sizes become uneven and the count is not predictable from a formula. Paragraph mode goes further and keeps headings with the text beneath them, which matters if your retriever will show the chunk to a human. Neither is better in the abstract. Fixed windows win on prose with no structure; structural splitting wins on anything with headings, lists or code fences, where a fixed window will happily cut a table in half.
What this tool does not model is the retrieval side. It will not tell you whether 400 characters is the right size for your corpus — only measured recall on your own questions can, and it usually says something different from what the defaults suggest.