Corpus Token Counter
Point at a folder and get an estimated token total, broken down by folder, by extension and by largest file.
0 files · 0 characters · one pass at $0.1 per 1M tokens costs $0.
- Files counted
- 0
- Files skipped
- 0
- Characters
- 0
- Estimated tokens
- 0
- Density
- —
- Cost of one pass
- $0
- Cost of ten passes
- $0
usage object on an actual API response: that is what you are billed on, and it includes chat-template wrapping this model cannot see. Only files with an extension on a text list are read; anything else, anything over 8 MB, anything with a null byte in it and anything past a 64 MB total for one run is skipped and listed below, so a total is never quietly short. Archives are not opened — a .zip is skipped like any other binary, because unzipping in the browser would need a library this page does not ship and a corpus counter that mis-reads an archive is worse than one that admits it cannot read one. The cost line is a single pass over every token; a re-embed, a second model, or a chunker with overlap all multiply it.What a corpus total is actually for
One number decides whether a retrieval project is a weekend or a quarter: how many tokens are in the pile. It sets the one-off embedding bill, it sets the cost of every re-embed when you change model, and — at the top end — it decides whether long-context stuffing is even in the conversation or whether retrieval is mandatory.
The breakdowns matter more than the total. Corpora are almost never evenly distributed: one folder of generated files, one vendored dependency, or a handful of enormous logs usually accounts for most of the count, and excluding them is a one-line change that halves the bill. Sorting by folder and by extension before you embed anything is the cheapest optimisation available, and the largest-files list is normally where the surprise is.
Three things to hold this number loosely for. It is an estimate, and it is at its worst on exactly the content a corpus is often made of — source code, minified JSON and CJK text all diverge from a character-class model by more than prose does; count one representative file through a real tokenizer and scale. It counts your files, not what you will send: chunking with overlap emits more text than the document contains, so the embedding bill exceeds the corpus total by the overlap fraction. And it says nothing about storage, which is a function of vector count and dimensionality rather than of tokens.