JSONL Dataset Token Counter
Paste or load a JSONL training file and get an estimated token count per example and for the whole file, with the training cost at your own rate.
Nothing is uploaded. Your dataset is read with FileReader inside this tab and never leaves it — there is no server call on this page, and the text is kept out of the URL as well, so the link you share carries your settings and not your content.
627 estimated tokens billed — 209 in the file, × 3 epochs.
- Non-empty lines
- 5
- Examples parsed
- 5
- Lines skipped
- 0
- Shapes handled
- messages[] × 3, prompt/completion × 1, instruction/input/output × 1
- Estimated tokens, one pass
- 209
- …of which is message overhead
- 44 (21.1%)
- Shortest example
- 28 tokens
- Median example
- 42 tokens
- Longest example
- 62 tokens
- Examples over 4,096 tokens
- none
- Cost per epoch
- $0.0017
Estimated, not tokenized. Token figures here come from an approximation running in your browser: no tokenizer vocabulary is downloaded, because a real one is megabytes and nothing on this page fetches anything. It imitates how a byte-level BPE splits text — words, digit groups, punctuation runs, whitespace runs — but it has no merge table, so treat it as a planning number. The authoritative count is the usage object on a real API response.
A fine-tuning bill is one number multiplied by two others: tokens in the file, epochs, and a price per million. The part people get wrong is the first one, and they get it wrong in a specific way — they count the text and forget the scaffolding. Every message in a chat-format example is wrapped by the trainer in role markers and separators before the model ever sees it, so a file of 5,000 short two-turn examples carries something like 40,000 tokens of pure structure. On short examples that overhead is not a rounding error; it can be a fifth of the file.
The shapes handled here are the ones that actually turn up in the wild: messages[] in the OpenAI chat format (including content given as an array of typed parts, which is joined before counting), conversations[] in the ShareGPT style, prompt/completion, the Alpaca instruction/input/output trio, bare input/output, and a plain text field. Anything else is listed above as a skipped line with the keys it did have, rather than being silently counted as zero — a dataset tool that quietly ignores a third of your file is worse than no tool.
What the total does not tell you is whether the dataset is any good. Token count is a cost input, not a quality signal, and the distribution matters more than the sum: a file whose median example is 200 tokens and whose longest is 12,000 is a file with a handful of examples about to be truncated mid-answer, which teaches the model to stop early. The shortest/median/longest rows above are there for that check. Run it before you pay for three epochs of it.