Reasoning Effort Settings: What the Knob Actually Does

There is no accuracy-per-effort-level table worth publishing, because the levels are not comparable between vendors, not stable between model releases, and not meaningful outside a specific task. What is worth publishing is exactly what the knob changes, what it fails to promise, and how to find your own number in three runs.

Two control styles

Providers have converged on two shapes for the same idea, and they fail in different ways.

If you have the choice, prefer the budget style for anything you intend to cost-model, because the unit is the same unit your invoice is in. Effort enums are more convenient and less auditable: two identical requests at “high” can differ by thousands of tokens, and a model update can shift the whole level without any change on your side.

What it does not guarantee

Three misreadings, in rough order of how much money they cost.

• It is not a cap. Anthropic’s documentation is explicit that the thinking budget is a target the model may not spend in full — and, on the other side, that the hard ceiling on the response as a whole remains max_tokens. Budget for the distribution, not the mean.
• It is not monotonic in accuracy. Higher effort is not reliably better; see overthinking for the published cases where it is worse. Treat the level as a parameter to tune, not a quality dial to turn up.
• It is not comparable across models. “High” on one vendor’s model and “high” on another’s share a word and nothing else. Any cross-model comparison has to be run at matched spend, not at matched label — which is the single most common flaw in informal comparisons you will read.

Calibrating on your own eval

You need three things before this is worth doing: fifty to two hundred examples of your real task, an automatic grader that returns a boolean per example, and the willingness to accept a number that disagrees with your intuition. The procedure is then mechanical.

const LEVELS = ["low", "medium", "high"];

for (const effort of LEVELS) {
  let correct = 0, reasoning = 0, output = 0, input = 0;
  const latencies = [];

  for (const ex of evalSet) {
    const t0 = Date.now();
    const r = await call({ ...ex.request, reasoning: { effort } });
    latencies.push(Date.now() - t0);

    const d = r.usage.completion_tokens_details ?? {};
    reasoning += d.reasoning_tokens ?? 0;
    output += r.usage.completion_tokens;   // includes reasoning
    input += r.usage.prompt_tokens;
    if (grade(ex, r)) correct++;
  }

  const cost = input * IN_RATE + output * OUT_RATE;
  latencies.sort((a, b) => a - b);

  report({
    effort,
    accuracy: correct / evalSet.length,
    costPerCorrect: cost / Math.max(correct, 1),
    reasoningShare: reasoning / output,
    p95ms: latencies[Math.floor(latencies.length * 0.95)],
  });
}

Two details in there are load-bearing. First, completion_tokens already includes the reasoning tokens on the OpenAI-shaped APIs, so adding them again double-counts your bill — a mistake that makes reasoning look twice as expensive as it is. Second, the headline number is cost per correct answer, not cost per call. A level that is thirty per cent dearer and lifts accuracy from 0.62 to 0.81 is cheaper on the metric that matters, and cost per call will never reveal that.

How many examples you need

Enough that the difference you are reading is not noise. Accuracy on an eval set is a binomial proportion, so its standard error is sqrt(p(1-p)/n). At n = 100 and p = 0.75 that is about 4.3 percentage points, and the standard error of the difference between two levels is larger still — around six points. So with a hundred examples, a five-point gap between medium and high is indistinguishable from nothing, and treating it as real is how teams end up paying indefinitely for an effort level that never helped.

Two ways out, both cheap. Use a paired comparison rather than two independent accuracies: run the same items at both levels and look only at the ones that changed answer, which removes the variance contributed by items neither level ever gets right. Or grow the set — at n = 400 the standard error halves. If neither is available, be explicit that you can only detect large effects, and act only on large effects.

Reading the result

You are looking for one of four shapes, and each has a clear action.

• Flat accuracy across all three levels. Your task does not need thinking. Run at the lowest level, or use a non-reasoning model entirely — the taxonomy of tasks where thinking changes nothing will usually tell you which of yours it was.
• A step up from low to medium and nothing after. The common case. Take medium; the extra spend at high is pure loss.
• Monotone rising with rising cost per correct answer. Accuracy is being bought, and you now have a business decision with real numbers on both sides rather than a preference.
• Accuracy falling at high effort. Real, documented, and not a bug in your harness — check the failure cases before you assume it is.

Record the reasoningShare figure too. If ninety per cent of your output tokens are invisible thinking, an effort reduction is the largest single cost lever available to you, larger than any prompt shortening you could do.

Keep the raw per-item results rather than only the summary. The summary tells you which level to pick; the per-item records tell you which items changed and in which direction, and that is the only view in which a level that improves the aggregate while breaking things you previously got right is visible at all. Store them somewhere you can re-read after the next model update, because the comparison you will want then is against this run, not against a number in a document.

Reasonable starting points

Until you have run the above: start at the lowest non-zero level for anything user-facing and interactive, because latency compounds with effort and a reader will abandon before an accuracy gain can help them. Start in the middle for batch work with a grader. Reserve the top level for tasks where you can name the cost of a wrong answer in currency — a mispriced quote, a bad migration, an escalation — since that is the only situation in which the arithmetic in the reasoning cost model comes out in favour of the top level.