Chunk Count Estimator
Takes a corpus through chunking to vectors to storage, including the tokens overlap makes you embed twice.
4 per document at 512 tokens with 64 overlapping. You will embed 12.0% more tokens than the corpus contains.
- Tokens per document
- 1,596
- Stride — chunk minus overlap
- 448 tokens
- Chunks per document
- 4
- Chunks in the corpus
- 40,000
- Corpus tokens
- 15,960,000
- Tokens actually embedded
- 17,880,000
- Overlap tax
- 12.0%
- Vector storage
- 117.2 MiB
- Stored chunk text
- 78.1 MiB
- Both together
- 195.3 MiB
The overlap tax nobody budgets for
Chunk count is the number every other line item in a retrieval system is multiplied by. It sets your embedding bill, your index memory, your storage, and how long the backfill takes. It is worth deriving rather than guessing, and the derivation has one term people miss.
Overlap is not free. If chunks are 512 tokens with 64 overlapping, the stride is 448, so a document is cut into ceil((tokens − 512) / 448) + 1 pieces and the overlapping tokens are embedded twice. That is about 14% more tokens than the corpus contains, 14% more vectors than the no-overlap case, and 14% more of everything downstream — paid once at backfill and again on every reindex. Push overlap to half the chunk size, which people do reach for when retrieval quality disappoints, and you are paying for the corpus twice.
The other lever is chunk size itself, and it moves the count inversely: halving the chunk roughly doubles the vectors. Small chunks retrieve more precisely and give the generation stage less context to work with; large chunks do the reverse and cost less to store. There is no universally right answer, which is why this tool computes consequences rather than recommending a size — pick the size from a retrieval evaluation on your own corpus, then come back here to find out what it costs.