Self-Consistency: Sampling Five Times and Voting

Ask the same question five times, take the answer that appears most often, discard the rest. It is the least clever technique in prompting and one of the few with a large, replicated effect — for a narrow class of task, at five times the price.

How it works

Self-consistency, introduced by Wang et al. (2023) in Self-Consistency Improves Chain of Thought Reasoning in Language Models, replaces greedy decoding with sampling. You generate k chains of thought at a temperature well above zero, extract the final answer from each, and take the majority.

The reasoning is that a hard problem has many valid derivations and many more invalid ones, but the invalid ones disagree with each other while the valid ones converge. Marginalising over reasoning paths and keeping the modal answer is therefore a cheap approximation to asking which answer the model believes most. Note the precondition, which is where most misapplications start: temperature must be non-zero, or all k samples are the same sample and you have bought nothing.

A second detail decides whether it works at all: the diversity has to be in the reasoning, not in the wording. Sampling hot enough to produce genuinely different derivations also makes each derivation slightly worse, so there is an interior optimum rather than a monotone knob, and the original work uses ordinary sampling settings rather than extreme ones. If your k samples differ only in phrasing and agree on every intermediate step, you have paid five times for one answer with better error bars on nothing.

What the paper reported

The reported gains were large on arithmetic and commonsense reasoning benchmarks — on the order of seventeen points on GSM8K and around eleven on SVAMP with the largest model they evaluated, with smaller but positive gains on several other sets. Those are their numbers on their models and benchmarks, published with the method; nothing on this page is a re-measurement.

Two structural details from the paper transfer better than the headline figures. The gains come from tasks with a discrete, comparable final answer — a number, a label, a multiple-choice letter — because that is what makes a vote possible. And accuracy against k saturates: their sweeps show most of the available benefit arriving in the first handful of samples, with the curve flattening well before the largest k they tried. That saturation is what makes the cost question decidable.

The break-even calculation

Self-consistency is worth it when the expected saving from fewer wrong answers exceeds the extra inference cost. Write it out:

worth it  iff  Δa · E  >  (k − 1) · c

  Δa  accuracy gained, in absolute points (measure this on your eval set)
  E   what one wrong answer costs you
  c   cost of one call
  k   samples

rearranged, the accuracy gain you need:   Δa > (k − 1) · c / E

Numbers make the shape obvious. Say a call costs $0.006 and you use k = 5, so the extra spend is $0.024 per question.

• A wrong answer costs six minutes of human review at a $40/hour loaded rate, so E = $4. You need Δa > 0.024/4 = 0.6 percentage points. Almost any real gain clears that.
• A wrong answer costs a $60 refund and it happens on 2% of traffic. E = $60. The threshold is 0.04 points. Not a decision — just do it.
• A wrong answer costs a mildly worse autocomplete suggestion, worth perhaps $0.02. The threshold is 120 percentage points, which is impossible. Never do it.

The lesson is that self-consistency is decided almost entirely by E, not by the technique’s effectiveness. High-stakes, low-volume, discrete-answer work is where it belongs; user-facing suggestions at scale is where it does not.

Run the same arithmetic against the alternatives before committing, because five samples of a mid-tier model is roughly the budget of one call to something considerably stronger, or of one call plus a verification pass, or of a retrieval step that removes the ambiguity causing the errors in the first place. Self-consistency competes with those, and it wins mainly when three things hold at once: the answer is discrete, the errors are reasoning slips rather than missing information, and a stronger model is not available at a price you like.

Latency is usually not the objection. The k samples are independent, so fanning them out concurrently makes wall-clock time the maximum of the calls rather than the sum — plus whatever your provider’s concurrency limit does to that at volume.

When the answers are not comparable

Majority voting needs an equality test. For free-form output there isn’t one, and three options remain, in increasing order of expense:

• Vote on an extracted field. If the answer contains a decision, a number or a citation id, vote on that and use one sample’s prose for the rest. Cheapest and usually sufficient.
• Cluster by embedding. Embed the k answers, cluster, and take a representative of the largest cluster. Adds an embedding call and a threshold you have to tune.
• Ask a model to pick. A judge call over the k candidates. This is no longer self-consistency — it is a best-of-n selection with a learned scorer, and it inherits the judge’s biases, including a well-known preference for longer answers.

The vote margin is free calibration

The most under-used output of self-consistency is not the winning answer, it is the distribution. A 5–0 vote and a 2–2–1 split arrive with identical confidence in the prose and very different reliability underneath.

So log the margin and route on it: unanimous goes straight through, a bare plurality goes to review or triggers a fallback to a stronger model. You have already paid for the samples; the agreement rate is a confidence signal you get for nothing, and it is better grounded than asking the model how sure it is.

Treat the margin as a relative signal rather than a probability. Five samples give you only a handful of possible margins, so it is a coarse instrument, and models agree confidently on wrong answers when the error is in a shared premise rather than in the arithmetic. Calibrate the routing threshold against your own labelled cases before you trust it, and re-check it whenever k or the temperature changes, since both move the distribution of margins.