How Many Tokens Are in a Book, a Codebase, a Year of Email

A token is not a word and not a character, so every figure on this page is a published word count multiplied by a stated ratio. The ratio is the assumption; change it for your tokenizer and every number below moves predictably.

The ratio, and how to measure your own

For ordinary English prose with a modern byte-pair-encoding tokenizer, the useful working figures are about 1.3 tokens per word and about four characters per token. Both are approximations that hold well across normal writing and break down on unusual text.

  tokens  ~=  words  * 1.3
  tokens  ~=  characters / 4

These two are consistent with each other for English prose, where the
average word plus its following space is about 5.2 characters:

  5.2 characters per word / 4 characters per token  =  1.3

Where the ratio moves, and by how much:

  English prose            1.25 - 1.35 tokens/word
  Technical / medical      1.4  - 1.7   (long rare words split into pieces)
  Source code              1.5  - 2.5   (punctuation and identifiers)
  Non-Latin scripts        2    - 4x worse per character
  Numbers and IDs          up to 1 token per 1-3 digits

Measure your own rather than trusting the default. It takes one line once you have a tokenizer, and the result is the only ratio that describes your text. There is more on what drives the differences in tokens per word and what a token is.

Books

  tokens  =  words * 1.3

  work                            words      tokens
  ---------------------------  ---------  ----------
  a typical business book         50,000      65,000
  a typical novel                 90,000     117,000
  The Great Gatsby                47,000      61,000
  Harry Potter and the
    Philosopher's Stone           77,000     100,000
  Moby-Dick                      206,000     268,000
  War and Peace                  587,000     763,000
  the King James Bible           783,000   1,018,000
  Shakespeare, complete works    885,000   1,150,000

Three consequences worth drawing out. An entire novel fits inside a 128,000-token context window with room to spare, which is why “summarise this book” became possible in one call. War and Peace does not fit in 128k but does fit in a million-token window. And the complete works of Shakespeare, at roughly 1.15 million tokens, is about the largest single artefact anyone routinely uses as a test of long context.

Fitting is not the same as working. Retrieval accuracy degrades well before the context limit — see lost in the middle and effective context length.

Codebases

Code tokenises worse than prose because punctuation, indentation and compound identifiers all split. Work from characters rather than from words.

  tokens  ~=  characters / 3.3        (code, not 4 as for prose)
  characters  ~=  lines * 40          (measure yours; 30-50 is typical)

  so:  tokens  ~=  lines * 12

Worked:

  a single 300-line file            300 lines  ->     3,600 tokens
  a small service, 10k lines         10,000    ->   120,000 tokens
  a medium application, 100k lines  100,000    -> 1,200,000 tokens
  a large monorepo, 1M lines      1,000,000    ->    12M tokens

Check the divisor on your own repository:

  find . -name '*.py' -not -path './.venv/*' | xargs wc -lc

The figure that matters is almost never the whole repository. Most line counts are dominated by lock files, generated clients, vendored dependencies, test fixtures and minified assets, none of which a model needs. Excluding those routinely removes 70 to 90 per cent of a repository’s lines, which turns an impossible context problem into an ordinary one. That selection is the actual work, and it is covered in what to put in a coding agent’s context and retrieval over code.

A year of email

  tokens  =  messages * words_per_message * 1.3

Assumptions, all adjustable:
  messages received per working day   50
  working days per year              250
  words per message                  150   (short business email)

  50 * 250                =  12,500 messages per year
  12,500 * 150            =  1,875,000 words
  1,875,000 * 1.3         =  2,437,500 tokens

At 200 messages a day, which is not unusual in some roles:

  200 * 250 * 150 * 1.3   =  9,750,000 tokens

Then the multiplier nobody accounts for: quoted reply chains. A thread
of 8 messages where each quotes all previous ones contains the first
message 8 times. Naive ingestion of a mailbox can inflate the true
token count by 3-5x.

The quoting effect is why deduplicating and stripping quoted text is the first step of any mailbox pipeline, and why a year of email is a retrieval problem rather than a context problem regardless of how large context windows get.

Meetings, documents and web pages

Speech, at a normal conversational rate of about 130 words per minute:

  1 hour of meeting   =  130 * 60          =  7,800 words   ~   10,100 tokens
  1 hour of podcast   =  same order        ~  10,000 tokens
  a working week of
    back-to-back calls (20 h)              ~ 200,000 tokens

Documents, at roughly 500 words per printed page of prose:

  a 1-page memo                            ~      650 tokens
  a 10-page report                         ~    6,500 tokens
  a 300-page technical manual              ~  195,000 tokens
  a 200-page annual report (dense tables)  ~  200,000+ tokens
                                              tables tokenise badly;
                                              measure rather than assume

Web pages, article text only after boilerplate removal:

  a news article, 800 words                ~    1,040 tokens
  a long-form feature, 4,000 words         ~    5,200 tokens
  a documentation page, 2,000 words        ~    3,000 tokens
                                              code samples raise the ratio

The script that reproduces all of it

Do not trust the ratios above for anything that matters. This measures the real number for your text and your tokenizer.

# pip install tiktoken
import sys, tiktoken

enc = tiktoken.get_encoding("o200k_base")   # or "cl100k_base"

text = sys.stdin.read()
tokens = len(enc.encode(text))
words = len(text.split())
chars = len(text)

print(f"characters      {chars:,}")
print(f"words           {words:,}")
print(f"tokens          {tokens:,}")
print(f"tokens/word     {tokens / max(words, 1):.3f}")
print(f"chars/token     {chars / max(tokens, 1):.3f}")

# Usage:
#   cat book.txt | python count.py
#   curl -s https://www.gutenberg.org/files/2701/2701-0.txt | python count.py

Two cautions on the script. It counts the text you give it, so strip front matter, licence headers and boilerplate first or you are measuring those. And the tokenizer name matters: encoding the same file with a 50,000-entry and a 200,000-entry vocabulary gives materially different totals, which is precisely the point of comparing tokenizers. For counting before a call rather than after, see counting tokens before sending.

What that costs to read

  cost  =  tokens / 1e6  *  price_per_million_input_tokens

At an input price of $3.00 per million tokens:

  a novel        (117,000 tokens)  =  0.117 * 3.00  =  $0.35
  War and Peace  (763,000 tokens)  =  0.763 * 3.00  =  $2.29
  a 100k-line codebase (1.2M)      =  1.200 * 3.00  =  $3.60
  a year of email (2.4M)           =  2.400 * 3.00  =  $7.31

At $0.15 per million, the same four are $0.018, $0.11, $0.18 and $0.37.

The arithmetic is trivial and the trap is not: these are the costs of reading each thing once. An agent that re-sends a growing conversation on every turn pays for the whole prefix again each time, so a ten-turn conversation over a large document can cost many times the single-pass figure. Prompt caching exists for exactly this and changes the arithmetic substantially — see what prompt caching actually saves and session token accounting.