The Cost Curve of Reasoning: Working Out Accuracy per Dollar

There is no universal accuracy-per-dollar curve for reasoning models, and anyone showing you one has drawn it on a task that is not yours. What generalises is the model that produces the curve. Here it is, with every input named and nothing filled in that you cannot check.

Cost per call is the wrong metric

Cost per call makes reasoning look indefensible. It is five to twenty times the fast model and the answer looks the same length. The metric that decides anything is cost per correct answer, because a wrong answer has no value and you paid for it anyway:

cost_per_correct = (in_tokens * in_rate + out_tokens * out_rate) / accuracy

where out_tokens INCLUDES hidden reasoning tokens, and accuracy is
measured on your own eval set with your own grader.

The division by accuracy does all the work. At 50% accuracy you are paying twice per useful answer; at 90% you are paying 1.11 times. A tier that costs three times more and lifts accuracy from 0.45 to 0.88 costs 1.53 times per correct answer, not three times — and that is the number to take to whoever owns the budget.

The five numbers you need

The first four come out of the harness in calibrating reasoning effort. The fifth comes from your business, and refusing to estimate it does not make it absent — it makes it implicit and usually wrong.

A worked example

The numbers below are illustrative placeholders, not measurements of any real model. They exist to show the shape of the arithmetic; replace every one of them before drawing a conclusion.

ILLUSTRATIVE ONLY — substitute your own measurements.

Assume: in_rate 0.50 / Mtok, out_rate 2.00 / Mtok, in_tokens 1,200.

tier        out_tok   cost/call    accuracy   cost per correct
----------------------------------------------------------------
fast            300   0.0012          0.52       0.00231
low           1,100   0.0028          0.71       0.00394
medium        3,400   0.0074          0.83       0.00892
high          9,800   0.0202          0.86       0.02349

Cost per correct answer rises monotonically here even though
accuracy also rises. On these assumptions the fast model is the
cheapest way to buy a correct answer — and that is the usual
result when V is small.

If your own table comes out like this one, cost per correct answer has told you what to do only if correct answers are all that matter and wrong ones are free. They rarely are, which is what the next section is for.

Two adjustments belong in the table before you trust it. If you use prompt caching, the effective input rate for the cached portion is lower, and since input tokens are constant across tiers this shifts the comparison slightly toward the cheaper end — the caching discount applies to the part that does not distinguish the tiers. And if any part of your workload is eligible for batch pricing, that discount applies to the tier you would otherwise have rejected on cost, which occasionally flips the ranking outright. Both are worth checking because both are structural rather than promotional: they change which row wins, not merely by how much.

Adding the value of being right

Bring in V, the net value of a correct answer over an incorrect one, and the comparison inverts for anything consequential. Expected value per call is accuracy × V − cost_per_call. Using the illustrative table above, the fast tier at 0.52 accuracy and the high tier at 0.86 differ by 0.34 in accuracy and by 0.019 in cost. So the high tier wins whenever:

0.34 * V  >  0.019        =>   V  >  0.056

Break-even at roughly six cents of value per correct answer.

Generally:   V_breakeven = (cost_hi - cost_lo) / (acc_hi - acc_lo)

Six cents. Almost any task a person will read the output of clears that by two or three orders of magnitude — a support answer, a code review, a contract clause. Which is why the honest summary of the whole cost debate is: reasoning is expensive per token and almost always cheap relative to the value of the decision, except in high-volume low-value work, where it is the wrong tool entirely.

That is also the formal version of the routing argument. If V varies across your traffic — and it does — then the optimal tier varies with it, per request, which is exactly what a router implements.

One cost that belongs in V and is almost always left out: human review time. If a wrong answer is caught by a person before it does damage, the cost of being wrong is not the damage, it is the review — and the review happens whether or not the answer was wrong. Raising accuracy to the point where you can drop the review is worth vastly more than the accuracy itself, and the break-even calculation looks entirely different on either side of that threshold. Work out where your threshold is before you optimise towards it, because the incremental accuracy either buys you that or buys you very little.

Three ways this goes wrong

• Forgetting that hidden tokens are billed. Estimate out_tokens from the visible answer and your model is wrong by the reasoning share, which is routinely above 80%. Read it from the usage object, never from the string length.
• Using mean cost where the distribution is skewed. Trace lengths have a long right tail. A mean-based forecast will under-predict a month in which the traffic got harder, and “the traffic got harder” is not a rare event.
• Treating accuracy as a fixed property. It is a property of a model, a prompt, a grader and a date. Re-run the table when any of those changes, and especially after a model update — the direction is not guaranteed, as the overthinking results show.

A last practical note: the evaluation itself costs money, and running four tiers across two hundred examples is a real invoice line. Budget for it explicitly rather than letting it appear as an anomaly, and treat it as a fixed cost against the monthly saving it identifies. On any workload large enough to be worth optimising, an evaluation that costs an afternoon and a few tens of currency units to run pays back in days — and the version of this exercise that never happens is almost always the one nobody was allowed to spend anything on.