Making a Model Abstain Instead of Guessing

Every team that has tried to make a model say “I don’t know” has discovered the same thing: it says it for a week, then reverts, or it says it constantly and becomes useless. Both outcomes have the same cause, and it is not the prompt.

Guessing is the rational move

Kalai, Nachum, Vempala and Zhang put the argument crisply in Why Language Models Hallucinate (2025). Consider a question the model would get right with probability q. Under binary grading — one point for correct, zero otherwise, which is how almost every benchmark and most human raters score — the expected score of guessing is q, and the expected score of abstaining is zero. For any q > 0, guess.

Now run reinforcement learning from human feedback against preferences collected under roughly that scoring, and against benchmarks scored exactly that way, over many iterations. You get a model that has been systematically trained out of the hedge. The paper’s framing is that hallucination persists after post-training not because alignment failed but because alignment succeeded at the objective it was given, and the objective contained an epidemic of binary graders.

This is why prompt-only attempts decay. You are asking, at inference time, for a behaviour that was penalised throughout training. It is possible — the distribution is not a wall — but you have to make the abstention worth something in the model’s local objective, and the only lever you have at inference is the prompt’s implied scoring rule.

Fix the scoring, then the prompt

The move the paper suggests, and it works as a prompting technique because it changes what “good answer” means inside the context: state a confidence threshold and the penalty explicitly.

Pick a threshold t — the confidence below which you would rather have nothing. The scoring rule that makes t the correct cutoff gives +1 for a correct answer, 0 for an abstention, and a penalty of t / (1 - t) for a wrong answer. At t = 0.75 the penalty is 3; at t = 0.9 it is 9. Put that arithmetic in the prompt rather than the word “careful”:

Answer only if you are more than 75% confident. Scoring for this task:
a correct answer is worth +1, "I don't know" is worth 0, and an
incorrect answer is worth -3. Answering when unsure loses points.

If you cannot meet the threshold, reply exactly:
INSUFFICIENT_EVIDENCE: <the one fact you would need>

Two things are doing the work here and both are load-bearing. The numbers make abstention a positive-expected-value action inside the frame the model is reasoning in. And the exact required token makes abstention machine-detectable, so you can measure it, route on it and count it — a refusal phrased freely in prose is invisible to your metrics.

The naming of the missing fact matters more than it looks. It converts a dead end into a retrieval query, and in a grounded system it is usually the thing that turns an abstention into an answer on the second pass.

Prompts that actually license a refusal

Beyond the scoring rule, a small number of framings reliably shift behaviour, and all of them work by removing the implicit demand for an answer:

• Make abstention an enumerated option, not an absence. If the output schema is an enum that includes UNKNOWN, the model is choosing among options rather than failing to produce one. Structured output is a much stronger lever here than instruction text.
• Separate extraction from answering. Ask first for the spans of evidence, then for the answer conditioned on them. An empty evidence list is a far easier thing for a model to produce than a refusal, and it makes the refusal a consequence rather than a decision.
• Say what happens next. “If you are unsure, say so and the question will be routed to a specialist” performs better than a bare prohibition, because it supplies a completion in which not answering is still helpful.
• Do not stack it with a persona demanding expertise. “You are a world-class expert” and “admit uncertainty” are in direct tension, and the persona usually wins.

There is published work on training this in rather than prompting it. Yin et al.’s SelfAware (2023) built a benchmark of genuinely unanswerable questions to test whether models know what they cannot know; R-Tuning (Zhang et al., 2024) fine-tunes on refusal-aware data, teaching the model to abstain on the subset it got wrong during training. Feng et al.’s 2024 survey collects the family. If you own a fine-tune, this is the durable version of everything above.

Abstaining on a signal, not a mood

The stronger architecture does not ask the model to decide. It computes a confidence signal outside the model and thresholds on it, which means the abstention policy is a number you can tune rather than a phrase you hope survives the next model update.

def answer_or_abstain(question, k=5, agree_threshold=0.6):
    """Sample k answers; abstain when they do not agree.
    The self-consistency signal, used as an abstention gate."""
    samples = [call_model(question, temperature=0.7) for _ in range(k)]
    clusters = cluster_by_entailment(samples)      # semantic, not string equality
    top = max(clusters, key=len)
    agreement = len(top) / k
    if agreement < agree_threshold:
        return {"abstain": True, "agreement": agreement, "samples": samples}
    return {"answer": top[0], "agreement": agreement}

This is the abstention face of the semantic-entropy and SelfCheckGPT line of work described on the calibration page: sampled answers that disagree are the signature of a fabrication, because a fabricated detail is drawn from a flat region of the distribution and a memorised one is not. It costs k times the tokens, which is the honest price of the most reliable knowledge-free detector currently published.

For a stronger guarantee, conformal prediction is the right frame. Given a calibration set, it converts any confidence score into a threshold with a distribution-free bound on the error rate among answered items — Quach et al.’s Conformal Language Modeling (2024) works this out for generation. The guarantee is only as good as the exchangeability assumption, which your traffic will violate the day the input distribution shifts, so re-calibrate on a schedule.

The risk–coverage curve

One plot decides whether an abstention policy is any good, and it is not “abstention rate before and after”. Sweep the threshold and plot:

• x-axis: coverage — the fraction of inputs the system answered rather than abstaining on, from 0 to 1.
• y-axis: risk — the error rate among the answered items only. This is the number that matters, and the one a bare accuracy figure destroys.

A useful confidence signal produces a curve that rises as coverage rises: the items you abstained on were disproportionately the ones you would have got wrong. A useless signal produces a flat line at the base error rate — you are abstaining at random, trading answers for nothing. That flat line is the outcome of most prompt-only attempts, and it is invisible unless you plot it this way.

Report the whole curve, then pick the operating point from the business: “at 80% coverage the error rate among answers is X” is a sentence a product owner can act on. And watch the other end — over-abstention is a real cost, and a model that hedges on questions it could answer is the failure described in refusals on legitimate work, arrived at from the opposite direction.