Cohere's Temperature Range and Default

Cohere documents temperature with a default of 0.3 and a range of 0.0 to 1.0. Both numbers differ from the provider most people arrive from, and copying a value across without adjusting it changes the setting rather than preserving it.

The documented values

Cohere’s Chat API reference documents temperature as a non-negative float defaulting to 0.3, with a documented minimum of 0.0 and a maximum of 1.0. It applies on both API versions.

{
  "model": "command-r-plus-08-2024",
  "message": "Name the three warehouse regions.",
  "temperature": 0.0
}

Why the range is narrower

OpenAI accepts temperature up to 2.0. Cohere caps at 1.0. This is the detail that matters when porting a request, and it cuts both ways.

A value above 1.0 sent to Cohere is out of the documented range — you get an error or a clamp, neither of which is what your code intended. And a value copied downward is not preserved either: 0.7 is a middling setting on a 0–2 scale and a high one on a 0–1 scale. The same number means something different, and the failure is silent, showing up as output that is more varied than the team expected and nobody quite knowing when it changed.

The narrower range is defensible on its own terms. The useful band of temperature is not linear: the interesting behaviour is concentrated below 1.0, and above about 1.2 most models degrade into incoherence rather than creativity. A cap at 1.0 removes a region that is mostly a trap. It does mean Cohere has no equivalent of the very high settings some people use deliberately for brainstorming, and p — nucleus sampling — is the parameter to reach for instead.

What the number does

The model produces a score for every token in its vocabulary. Temperature divides those scores before they are turned into probabilities. Dividing by a number below 1 spreads them apart, which makes the already-likely token more likely still; dividing by a number above 1 compresses them, flattening the distribution so unlikely tokens get a real share.

Two consequences follow that are easy to state and often forgotten. Temperature never adds a token that had a score of effectively zero — it re-weights what was already there, so a high temperature makes the model choose worse, not choose differently in kind. And the effect compounds across a long output: one slightly unusual token changes the context for every token after it, which is why the difference between 0.3 and 0.8 is barely visible in a ten-token answer and dramatic in a thousand-token one.

A concrete illustration of the compounding. Suppose the next token is “yes” with probability 0.9 and “no” with 0.1. At temperature 0 you get “yes” every time; at a moderate temperature you get it most of the time; and over a hundred such decisions in a long answer, a per-token deviation rate of even a few percent means the majority of long outputs contain at least one choice the model would not have made at temperature 0. That is the honest picture: temperature does not usually change the answer, it changes how often something in the answer differs — and long outputs give it far more opportunities than short ones.

Choosing a value

• 0.0 for extraction, classification and structured output. There is one right answer and no value in variety. Pair it with response_format rather than relying on temperature alone to keep the shape stable.
• 0.3, the default, for grounded question answering. Cohere chose it for the RAG-shaped work Command is built for, and it is low enough that the answer stays close to the documents.
• 0.7 to 1.0 for drafting and ideation, where you want several different attempts and will pick one.
• Not as a fix for a bad answer. Raising temperature because the output is wrong produces a different wrong output. The problem is almost always the prompt or the retrieved documents.

The other two sampling parameters

Temperature is one of three knobs on the sampler, and Cohere names the other two p and k rather than top_p and top_k — a small naming difference that breaks a request copied from another provider’s example. Both are documented in the Chat API reference alongside temperature.

• p, nucleus sampling. Keep the smallest set of tokens whose probabilities sum to p, and sample from that set only. It is adaptive: where the model is confident the set is one or two tokens, and where it is uncertain the set is wide. That adaptiveness is why it is generally the better lever than temperature for controlling how far the output can wander.
• k, top-k sampling. Keep the k highest-scoring tokens regardless of their probabilities. Fixed-width, and therefore blunt in exactly the place p is not: a k of 40 admits 39 bad options when the model is certain, and cuts off good ones when it is not. Cohere documents a disabled default, which is the right default.

The interaction is where people go wrong. Temperature reshapes the distribution; p and k truncate it. They apply together, so setting a high temperature and a tight p is two instructions that partly cancel — you have flattened the distribution and then thrown away the part you flattened it to reach. The result is not “creative but controlled”; it is a setting whose effect is hard to predict and harder to explain to whoever inherits it.

The practical advice is to move one and leave the others at their documented defaults. If output is too repetitive, raise temperature. If output is occasionally incoherent, lower p. Changing both in the same experiment means you learn nothing from either, which is the usual reason a team ends up with a sampling configuration nobody can justify.

Zero is not determinism

temperature: 0 means always take the highest-scoring token. It does not mean the same request returns the same bytes every time, and treating it as a guarantee will eventually cost you a debugging session.

Floating-point arithmetic on a GPU is not associative, so the order in which a batch is summed can change the last bits of a score. When two tokens are nearly tied, that is enough to flip which one wins, and every token after it diverges. Batch composition varies with other people’s traffic, which you do not control. Serving stacks are also upgraded, and an alias like command-r-plus can point at a different snapshot than it did last month — which is a separate argument for pinning a dated version.

So temperature 0 buys you a large reduction in variance, not the absence of it. If your test suite asserts on exact model output, it will go red one morning for reasons that have nothing to do with your code. Assert on properties — the JSON parses, the required fields are present, the classification is in the allowed set — and the suite tests what you actually care about.