Llama's Context Window Counted in Tokens: Fitting a Document

“Will this document fit?” is a question about tokens, and you have a word count. The conversion is not a constant — it is a property of your text — but the arithmetic around it is fixed, and getting the arithmetic right matters more than getting the ratio precise.

Why there is no single ratio

A tokenizer maps character sequences to entries in a fixed vocabulary. Llama 3’s has 128,256 entries, so common English words are often a single token and rarer strings are assembled from several. The tokens-per-word ratio for a piece of text is therefore a function of what is in it:

• Plain English prose — the best case. Most words are in the vocabulary whole.
• Technical and medical vocabulary — worse. Long rare words split into three or four pieces.
• Code and markup — much worse per “word”, because punctuation, indentation and identifiers like getUserById fragment. Word count is close to meaningless here; count characters or count tokens.
• Numbers and IDs — Llama 3’s pre-tokenizer groups digits in runs of up to three, so a UUID or a long run of identifiers costs far more than its word count suggests.
• Languages other than English — usually worse, and for scripts poorly covered by the vocabulary, dramatically so.

Any single number quoted as “tokens per word for Llama” is an average over somebody else’s corpus. Use it to sanity-check an order of magnitude, never to decide whether something fits.

If you are going to estimate from something, estimate from characters rather than words. A word is not a well-defined unit — splitting on whitespace counts a hyphenated compound as one and a snippet of code as almost none — whereas characters are unambiguous and correlate more steadily with token count across text types. Characters per token also degrades more gracefully: it is a ratio that moves smoothly with content, where tokens per word can double between two documents that look similar in a word processor.

The one published anchor

The widely cited rule of thumb — roughly four characters per token, or about three-quarters of a word per token for English — comes from OpenAI’s help documentation on counting tokens, and it describes OpenAI’s tokenizers, not Meta’s. It is quoted here because it is the only figure of this kind published by a model author, and because Llama 3’s byte-level BPE with a 128,256-entry vocabulary is a design of the same family and comparable scale, so the neighbourhood transfers even though the exact number does not.

Measuring your own ratio

Take a representative sample of the actual text you will send — not a clean paragraph, but a real document with its headings, tables and identifiers intact — and run it through the model’s own tokenizer:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

text  = open("sample.txt", encoding="utf-8").read()
words = len(text.split())
chars = len(text)
toks  = len(tok.encode(text, add_special_tokens=False))

print(f"words:            {words}")
print(f"tokens:           {toks}")
print(f"tokens per word:  {toks / words:.3f}")
print(f"chars per token:  {chars / toks:.3f}")

Run it over several samples and take the worst ratio, not the mean. A budget built on an average overflows on half your documents; a budget built on the worst case wastes a little context and never fails. Call that worst-case figure r for the rest of this page.

Re-measure r when the model family changes and when the content changes. A ratio derived from clean English articles does not survive the day somebody starts uploading spreadsheets exported as text, and the failure arrives as a wave of context-length errors rather than as a gradual drift.

The budget, term by term

The context window is shared by everything in the request and everything the model generates. Writing it out:

served_window
  - system_prompt_tokens
  - template_overhead
  - conversation_history_tokens
  - reserved_output_tokens
  = tokens available for the document

document_tokens ≈ document_words x r

Each term needs naming, because the ones people forget are the ones that cause the failure:

• served_window — what your endpoint enforces, not what the model card documents. A self-hosted server’s --max-model-len is frequently a fraction of the checkpoint’s maximum, for the memory reasons in the KV cache arithmetic.
• template_overhead — the special tokens the chat template wraps around every message: a header, a role, and a terminator per turn. Small per message, and not small across a long conversation. Measured properly by tokenizing with apply_chat_template rather than by tokenizing your strings.
• reserved_output_tokens — the most commonly omitted term. Output comes out of the same budget, so if you want a 1,000-token answer you must leave 1,000 tokens unused. Omitting it is what produces the context-length error after the prompt has already been accepted as valid.
• A safety margin. Ten per cent is a reasonable default, because r is an estimate and a document can be atypical.

A worked example

Assumptions, all of them stated and all of them yours to replace: a server started with --max-model-len 32768; a system prompt you measured at 400 tokens; a single-turn request, so no history; a wish for answers up to 1,000 tokens; template overhead of about 30 tokens; and a worst-case measured ratio of r = 1.35 tokens per word for your documents.

available = 32768 - 400 - 30 - 1000        = 31,338 tokens
with 10% margin                            ≈ 28,200 tokens
max words                = 28,200 / 1.35   ≈ 20,800 words

Roughly twenty thousand words, which at about 500 words to a page is something like forty pages of prose. Change the ratio to 1.6 for technical text with many identifiers and the same window holds about 17,600 words. Change the served window to 8,192 — a very common local default — and it holds under 5,000.

Two habits make this arithmetic reliable in production. Compute the budget rather than hardcoding it, so that changing --max-model-len or the system prompt cannot invalidate a constant somewhere else. And check the real count before sending: the estimate decides how to chunk, the tokenizer decides whether to send. Estimating is for planning; counting is for the request.

When it does not fit

The arithmetic tells you the document is too big; it does not tell you what to do next. Three approaches, with the cost of each stated, because the choice is usually a cost decision rather than a quality one.

• Retrieve instead of paste. Index the document, embed the question, send the handful of passages that match. Cheapest by a wide margin and usually the best answer as well, because the model is reading a page rather than a book. Fails when the task genuinely needs the whole document — a global summary, a consistency check across sections.
• Chunk and reduce. Split into pieces that fit, process each, then combine the results in a second pass. The cost is that the fixed overhead is paid per chunk: the system prompt and the instruction are re-sent every time, so ten chunks means ten copies of your preamble in the bill. Size the chunks as large as the budget allows, not as small as is convenient.
• Use a bigger window. Raise --max-model-len if memory allows, or move to an endpoint that serves more. The cost is memory and concurrency locally, and price and prefill latency everywhere.

Two details make chunking work better than it usually does. Split on structure — sections, paragraphs, function boundaries — rather than at a fixed token count, so that no chunk begins mid-sentence; and overlap adjacent chunks by a few hundred tokens so that a fact spanning a boundary appears whole in at least one of them. Both are cheap, and both remove a class of wrong answer that looks like a model failure and is a splitting failure.