Skip to content
Multigrid

Tokens, tokenization & context windows

The unit you are billed in, the limit you keep hitting, and the arithmetic that connects the two.

A token is the only unit that matters commercially. Prices are quoted in it, context limits are counted in it, rate limits are enforced in it, and every truncation bug you will ever debug is a disagreement about how many of them a piece of text contains. It is also the one unit nobody can eyeball: the same sentence is a different number of tokens in two model families, and a different number again once it has been wrapped in a chat template you never see.

These pages work from the tokenizer outward. What a token is, how the algorithm that produces it works, why the count differs from your estimate and from your invoice, what a context window really constrains, and how to design a prompt backwards from a cost ceiling. Where a claim needs a number, the number comes from a published source and is named.

What Is a Token? The Unit You’re Actually Billed In

What a token is at the level the tokenizer defines it, and the arithmetic that turns a token count into a number on an invoice.

5 min read

How Many Tokens Is a Word? Real Ratios by Language

Where the 0.75-words-per-token rule of thumb comes from, the conditions under which it holds, and how to get the ratio for your own text.

5 min read

Byte-Pair Encoding Explained by Building One

A working BPE trainer in about forty lines, then the four things production tokenizers add on top of it.

5 min read

Why Tokenizers Charge Non-English Users More

The mechanism that makes identical content cost several times more in some languages, what the published research found, and what you can do about it.

5 min read

Context Window vs Max Output Tokens: Not the Same Number

Two independent limits that produce nearly identical-looking 400 responses, and the one formula that keeps you inside both.

5 min read

What a 1M-Token Context Window Is Actually Good For

The jobs very long context genuinely does better than retrieval, the jobs it does worse, and the cost arithmetic that decides between them.

5 min read

Lost in the Middle: Why Long Context Degrades

The published finding that retrieval accuracy depends on where in the prompt the answer sits, why it happens, and a harness for checking it on your own model.

5 min read

Effective Context Length vs Advertised Context Length

What the advertised number is actually a property of, what the published long-context benchmarks measure instead, and how to find the knee for your own task.

5 min read

Prompt Tokens, Completion Tokens and Why They’re Priced Apart

The hardware reason a completion token costs several times what a prompt token costs, worked through with published device specifications.

5 min read

Cached Tokens: The Cheapest Tokens You’ll Ever Buy

What a prompt cache actually stores, the break-even algebra for the multipliers vendors publish, and the eight things that silently destroy a hit rate.

5 min read

Reasoning Tokens: The Invisible Line on Your Bill

Where hidden thinking tokens appear in the usage object, the empty-response failure they cause, and how to decide when they earn their cost.

5 min read

How to Count Tokens Before You Send a Request

Three ways to get an exact count client-side, what each one misses, and where a cheap estimate is good enough.

6 min read

Truncation Strategies When Your Prompt Won’t Fit

Five ways to make an oversized prompt fit, ranked by what each one destroys, plus the invariants that must survive all of them.

6 min read

Chunking Text: Sizes, Overlaps and the Trade-offs

Why the right chunk is defined by answerability rather than by a token count, and the evaluation loop that finds the size for your corpus.

5 min read

Sliding Window Attention and Streaming Long Documents

How a bounded attention window buys unbounded streaming, what it costs in recall, and the published designs that make it work.

5 min read

Tokenizer Differences Between Model Families

What differs between the major tokenizer families, why per-token prices are not comparable until you normalise, and a script that builds the comparison on your own text.

5 min read

Special Tokens, Chat Templates and Silent Formatting Bugs

The scaffolding wrapped around your messages before the model sees them, and the five formatting bugs that make a competent model look incompetent.

6 min read

Why Token Counts Differ Between Your Count and the Bill

Eight places tokens enter a request without passing through your counter, and a reconciliation procedure that finds which one is yours.

5 min read

Context Compression: Making 100k Tokens Fit in 10k

Five ways to shrink a prompt, what each destroys, and the break-even arithmetic against the two cheaper alternatives people skip.

5 min read

Token Budgets: Designing a Prompt Around a Cost Ceiling

Start from the cents you can afford per request and derive the prompt, rather than writing the prompt and discovering the bill.

6 min read

Other topics

Tokens, tokenization & context windows · Multigrid