Prompting for Refusals: Making a Model Say "I Don't Know"

“Say you don’t know if you don’t know” is the instruction most often added and least often obeyed. The reason is structural, and it means the fix is not a better sentence.

Why the default is to answer

A model returns a distribution over the next token given the context. Following a question, the overwhelmingly high-probability continuations are answers, because that is what follows questions in essentially all of the training data. “I don’t know” is a low-probability continuation that your instruction has to lift above every plausible answer, at every step.

Post-training pushes the same way: helpfulness is rewarded, and a fluent wrong answer looks more helpful than a refusal to anyone skimming. And there is no internal “unknown” state to consult — the model has a distribution, not a knowledge base with a miss. Nothing about that says abstention is unobtainable. It says abstention has to be designed rather than requested.

Give abstention an output

The first move is to stop treating abstention as the absence of an answer and make it a value in the contract:

{
  "answer": string | null,
  "source_id": string | null,
  "status": "answered" | "not_in_documents" | "ambiguous_question"
}

Rules given to the model:
  - Every sentence in "answer" must be supported by a <doc> in this prompt.
  - If no document supports an answer, set answer=null, status="not_in_documents".
  - If the question is answerable but underspecified, use "ambiguous_question"
    and put the clarifying question in "answer".
  - Answering with status="not_in_documents" is a correct outcome, not a failure.

Three things changed. There is now a specific token sequence for abstention, so it is one identifiable continuation rather than an open-ended act of self-restraint. Your code can branch on it. And the last line matters more than it looks: prompts routinely imply that abstaining is a failure state, which is a strange thing to do to a system trained to be helpful.

Separating not in the documents from ambiguous question is worth the extra enum value. They need different product responses — one escalates, one asks a follow-up — and merging them into a single “I don’t know” throws away the distinction the user cares about.

Choose the literal deliberately. It should be a string that cannot occur inside a legitimate answer, short enough to be cheap, and stable enough to grep for across a year of logs — NOT_IN_DOCUMENTS rather than a sentence, which the model will paraphrase. Then test equality rather than sentiment: a check that looks for “I don’t know” anywhere in the response will happily match an answer beginning “I don’t know exactly, but” and then invents the rest.

Make it evidence-backed and checkable

The strongest version of this pattern does not ask the model to judge its own knowledge. It requires evidence and then verifies the evidence exists:

• Give every retrieved chunk a stable id and require source_id on every answer.
• After generation, check the id exists and that the quoted span actually appears in that chunk. A fabricated citation is now a caught error rather than a convincing one.
• On a failed check, convert the response to an abstention rather than retrying into a better-looking fabrication.

This is the whole trick: you have replaced an unobservable question (does the model know?) with an observable one (is there a span that supports this?). Note the honest limit — it works for answer-from-provided-context tasks. For open-domain questions with no documents there is nothing to verify against, and you are back to calibration.

One more detail that is easy to miss: if you use few-shot examples, include abstaining examples. A block of five demonstrations that all answer teaches the label space, and that label space contains no abstention. This follows directly from what demonstrations are known to convey.

Measure both directions

Abstention prompts fail in two directions and almost everyone only measures one. Build two sets:

• Answerable set — questions whose answer is genuinely in the provided documents. Metric: false abstention rate.
• Unanswerable set — questions whose answer is absent, plus the nastier variants: an answer that is nearly there, an answer contradicted by the document, a question about a neighbouring topic.

Report both. A prompt that abstains on 100% of unanswerable questions and 40% of answerable ones is a worse product than the version you were complaining about, and the single-number version of this evaluation will call it an improvement. Every hardening of the abstention instruction should be re-checked against the answerable set — that is where the cost is paid.

The two-set design also catches a third outcome that a single set hides: a confidently wrong answer to an unanswerable question is a different bug from a confidently wrong answer to an answerable one. The first is an abstention failure and is fixed by evidence requirements. The second is a retrieval or capability failure and is not fixed by any amount of abstention wording. Tag them separately, or you will spend a fortnight tuning the wrong instruction.

Better signals than asking

Asking the model for a confidence number is the weakest available option: verbalised confidence tends to cluster at round, high values. Kadavath et al. (2022), Language Models (Mostly) Know What They Know, is the useful reference here — they found models could be prompted to predict the probability that their own answer is correct with reasonable calibration on some tasks, improving with scale, which is a more specific and more limited claim than “models know when they are unsure”.

Three signals that need no self-report:

• Sample agreement. Ask five times at non-zero temperature and look at the spread. Disagreement is a strong uncertainty signal and it costs you nothing beyond the samples.
• Token log-probabilities, where the API exposes them. The probability assigned to the decisive token is a usable threshold for routing to review.
• Retrieval score. If the best chunk is below your similarity threshold, abstain in code before the model ever sees the question. The cheapest refusal is the one that never becomes a generation.

Finally, give the abstention somewhere to go. If “I don’t know” is a dead end in the interface, someone will eventually remove the instruction to make the demo look better. Route it to a search, a human, or a specific next question, and abstention becomes a feature people defend rather than a gap they patch.