Cohere's Tokenizer and Vocabulary Size

Cohere’s Command models use a byte-pair encoding tokenizer with a vocabulary of roughly 255,000 tokens — several times the size of the vocabularies most open models ship with, and the reason a token count from another provider’s counter is not a usable estimate here.

The size, and where it comes from

The Command R generation is documented with a vocabulary of about 255,000 tokens, and Cohere publishes the tokenizer for each model rather than describing it in prose. The models endpoint returns a tokenizer_url per model pointing at a Hugging Face-format tokenizer JSON:

curl -s https://api.cohere.com/v1/models/command-r-plus-08-2024 \
  -H "Authorization: Bearer $CO_API_KEY" | jq '{name, tokenizer_url}'

{
  "name": "command-r-plus-08-2024",
  "tokenizer_url": "https://storage.googleapis.com/cohere-public/tokenizers/command-r-plus-08-2024.json"
}

That file is the authority, and it is worth stressing why the URL is per-model rather than global: different Command generations do not necessarily share a tokenizer, so “Cohere’s tokenizer” is not one object. Cohere’s tokens and tokenizers documentation covers the current set.

What a large vocabulary buys

A tokenizer’s vocabulary is a budget. Every entry spent on one common sequence is an entry not spent on another, and the size decides how many languages can be represented efficiently at once.

With a 30,000-token vocabulary, English words are mostly single tokens and everything else is assembled from fragments — a Japanese sentence, a Hindi one, or a chemical name can cost two or three times as many tokens as its English equivalent for the same meaning. Since context, latency and price are all denominated in tokens, that is a direct tax on non-English use.

Command was built for multilingual work, and a quarter-million entries is what that costs. The trade is real on both sides: the embedding and output layers scale with vocabulary size, so a large vocabulary spends parameters and memory that a smaller one keeps. The benefit is that the 128,000-token context window holds meaningfully more non-English text than the same number would on a model with a small vocabulary — the window is the same, the tokens buy more.

Counting tokens without generating

Cohere exposes tokenization directly, so you never have to estimate. The endpoint requires a model, because the answer depends on it:

curl https://api.cohere.com/v1/tokenize \
  -H "Authorization: Bearer $CO_API_KEY" \
  -H "content-type: application/json" \
  -d '{"text": "De fiets staat in Utrecht.", "model": "command-r-plus-08-2024"}'

{
  "tokens": [4568, 22090, 15310, 1671, 41290, 21],
  "token_strings": ["De", " fiets", " staat", " in", " Utrecht", "."]
}

token_strings is the field to look at when something is behaving oddly — it shows you exactly where the boundaries fell, which is how you discover that an identifier you thought was atomic is being split into five pieces, or that a stop sequence you chose is not a clean boundary. There is a matching /v1/detokenize for the other direction, and both exist in v2.

The endpoint is a network call and is rate limited like any other, which decides where it belongs in your architecture. Tokenizing one prompt before one generation is fine and adds a round trip you probably will not notice. Tokenizing every chunk of a hundred-thousand-document corpus is an abuse of it: the job will be slow, it will hit limits, and it will fail partway through with half a corpus indexed.

For local counting with no network call, load the tokenizer file from tokenizer_url into any Hugging Face-compatible tokenizer library. That is the right approach for anything counting in a loop — chunking a corpus, enforcing a per-request budget — because a round trip per chunk is slow and rate-limited.

Counts do not transfer between providers

This is the practical failure this page exists to prevent. A chunking pipeline built against one provider’s tokenizer and pointed at Cohere will not produce the chunk sizes you designed, and the direction of the error is not predictable from the vocabulary size alone — it depends on the text.

The rule that holds: count with the tokenizer of the model you are about to call, per model, and treat any cross-provider token count as a rough estimate with no guarantees. If you are sizing a retrieval chunk to fit a budget, leave headroom rather than aiming at the limit — the cost of a chunk that is 15% smaller than optimal is negligible next to the cost of a request rejected for being over the documented context length.

Estimating when you cannot tokenize

Sometimes you need a number before you have the text — sizing a budget in a design document, deciding whether a corpus will fit, putting a character limit on a form field. The usual rule of thumb for English is about four characters per token, and it is worth being explicit about when that rule holds and when it collapses.

• Prose in a well-represented language is where the heuristic works. Common words are single tokens and the average lands near the rule.
• Code is worse than prose. Indentation, punctuation, camel-case identifiers and operators fragment heavily. A minified bundle or a base64 blob is close to worst case — a long random string approaches one token per two or three characters and can approach one per character.
• Structured data is mostly punctuation. A JSON document spends a large fraction of its tokens on braces, quotes and commas. Passing records as JSON rather than as compact lines can double the token cost of the same information, which matters when documents are resent every turn.
• Languages without spaces — Chinese, Japanese, Thai — do not follow a characters-per-token rule at all, and Command’s large vocabulary is precisely what makes them cheaper here than on a small-vocabulary model. Estimate these by tokenizing a sample, not by scaling an English ratio.

The defensible method when the real tokenizer is not available: take a representative sample of your actual content, tokenize it once through /v1/tokenize, and derive your own characters-per-token ratio for that content type. One measured ratio for your own corpus beats a remembered rule of thumb, and it takes a single request to get. Then leave headroom, because the estimate is an average and the request that fails will be the outlier.

Special tokens are in there too

The vocabulary contains more than text fragments. Command’s chat template uses explicit turn delimiters — <|START_OF_TURN_TOKEN|>, <|USER_TOKEN|>, <|CHATBOT_TOKEN|>, <|END_OF_TURN_TOKEN|> and others — and these are single tokens in the vocabulary rather than sequences of characters.

Two things follow. Every conversation turn costs a handful of tokens beyond its visible content, which is invisible in your own counting and very visible in a long history. And when you are running open-weights Command locally and assembling the template yourself, the delimiters must be tokenized as special tokens, not as literal text — get that wrong and the model sees the punctuation of a turn marker rather than the marker, which produces output that is subtly wrong in a way no error message explains.