Question Generation and Grading Prompts

Generated questions fail in two ways that look like success: they are answerable from general knowledge without reading the source, and their difficulty is a matter of how the question is phrased rather than how much work the answer takes. Both are fixable, and the second fix is a control you can run.

Difficulty defined by work, not wording

“Hard” in a generated question usually means longer, vaguer, or full of subordinate clauses. That is difficulty of comprehension, not of the task, and it produces questions that are annoying rather than discriminating.

Define the levels by the retrieval and reasoning steps the answer requires. Then the level is a claim about the question that can be checked by looking at the supporting spans.

The generation prompt

Generate {{n}} questions from <source>.

Every question must be answerable from <source> alone, and must NOT be
answerable by a well-informed reader who has not seen <source>. If the answer
is general knowledge, the question is about the world and not about this
source; do not write it.

Difficulty is defined by the work the answer requires, not by the wording:
  level_1  The answer is one contiguous span in <source>.
  level_2  The answer requires combining two spans that are not adjacent.
  level_3  The answer requires applying a rule, rate or figure from <source>
           to a case <source> does not state but does determine.

Produce {{d1}} at level_1, {{d2}} at level_2, {{d3}} at level_3.

For each question return:
{"question": "...",
 "answer": "<the shortest complete answer>",
 "level": 1 | 2 | 3,
 "support": ["<verbatim span from <source>>", "..."],
 "step": "<for level 2 and 3: what you did with the spans to reach the answer>",
 "distractors": [{"wrong": "...", "why_tempting": "...", "why_wrong": "..."}]}

Rules:
- Every span in "support" must be a verbatim substring of <source>.
- level_1 has exactly one span. level_2 and level_3 have at least two, and
  they must not be adjacent sentences.
- Do not write a question whose correct answer is "the source does not say".
- Do not write two questions with the same answer.
- Do not use the source's exact phrasing in the question stem; a question a
  reader can answer by string-matching tests nothing.
- If <source> cannot support {{d3}} questions at level_3, produce fewer and
  say so in "shortfall". Do not manufacture difficulty with vague wording.

<source>
{{source}}
</source>

The distractors are worth the tokens even when you do not need multiple choice. Writing a plausible wrong answer forces the model to locate the near-miss in the source, and a question with no plausible distractor is usually a question with no discriminating power. Where a distractor turns out to be defensible, you have found an ambiguity in the source.

The ban on reusing the source’s phrasing in the stem addresses the most common quality problem: a question that is the answer sentence with one noun replaced by a question word. It is answerable by anyone with a text search and it tests nothing.

The no-source control

This is the check that makes the “not answerable from general knowledge” requirement more than a hope. Run every generated question a second time with the source removed.

Answer the question. If you do not know, say exactly: UNKNOWN.
Do not guess. Do not reason from what is likely.

Question: {{question}}

1. Run each generated question through the control call, with no source in the context and no conversation history.
2. Grade the control answer against the key with the grading prompt below.
3. Any question the control answers correctly is testing prior knowledge, not the source. Discard it — regardless of how good it looks.
4. Run the control three times at a non-zero temperature rather than once. A question answerable one time in three is still leaking.

Expect this to remove a meaningful share of questions from any source about a well-documented subject, and to remove almost none from an internal policy document. That difference is itself informative: if your control discards most of a set, the source is mostly restating public knowledge and a comprehension test over it will not measure comprehension.

The grading prompt

Grade <response> against <key>.

<key> contains the reference answer and the spans from <source> that support
it. <source> is included so you can check the key.

Return:
{"verdict": "correct" | "partially_correct" | "incorrect" | "key_is_wrong",
 "missing": ["<element of the key answer the response does not contain>"],
 "incorrect_additions": ["<claim in the response that <source> contradicts>"],
 "quote": "<the span from <source> that decides this, verbatim>"}

Rules:
- Grade content, not wording. A response that says the same thing in different
  words is correct. A response in the wrong format is correct if the content
  is right; format is not your concern.
- "partially_correct" requires at least one element of the key answer present
  and at least one missing. It is not a hedge for uncertainty.
- Extra correct information does not lower the grade. Extra information that
  <source> contradicts goes in "incorrect_additions" and makes the verdict
  at best "partially_correct".
- Use "key_is_wrong" when <response> is supported by <source> and <key> is
  not. Quote the span that shows it.
- Every verdict requires a "quote" from <source>.

<key>{{key}}</key>
<source>{{source}}</source>
<response>{{response}}</response>

Letting the grader disagree with the key

The key_is_wrong verdict is the part that most graders omit, and it exists because generated answer keys contain errors. A grader with no way to say so must mark a correct response wrong, and the error is then invisible: it looks like a student mistake, in a category where mistakes are expected.

The rate at which it fires is a direct quality measure of your generator. A few percent is normal on complex sources. Ten percent means the generation prompt is producing keys it cannot support, and the fix is upstream — usually tightening the support span requirements rather than touching the grader.

Keep grading and generation in separate calls with separate contexts. A single call that generates and grades has no independent view of the key, and where you have more than one model available, grading with a different one removes the self-preference effect described in judge bias. The rubric discipline that makes such grades reproducible across graders is in rubrics two judges agree on.

One paragraph, three levels

A short internal policy source, and one question at each level, so the difference between the levels is visible as a difference in the supporting spans rather than in the phrasing.

-- source -------------------------------------------------------------------
"Expenses are reimbursed at the standard rate of 0.23 EUR per kilometre for
 journeys made in a private vehicle. Claims must be submitted within 60 days
 of the journey. Claims over 500 EUR require a director's approval before
 submission. The standard rate does not apply to journeys between an
 employee's home and their normal place of work."

-- level 1 ------------------------------------------------------------------
{"question": "What rate applies to journeys made in a private vehicle?",
 "answer": "0.23 EUR per kilometre",
 "level": 1,
 "support": ["reimbursed at the standard rate of 0.23 EUR per kilometre"],
 "distractors": [{"wrong": "0.23 EUR per mile",
                  "why_tempting": "the unit is the only thing that changes",
                  "why_wrong": "the source says kilometre"}]}

-- level 2 ------------------------------------------------------------------
{"question": "A claim is submitted 50 days after the journey and comes to
              620 EUR. What else was required before it was submitted?",
 "answer": "A director's approval",
 "level": 2,
 "support": ["Claims must be submitted within 60 days of the journey",
             "Claims over 500 EUR require a director's approval before
              submission"],
 "step": "The 50-day detail satisfies the first span so it is not the
          blocker; the 620 EUR figure crosses the threshold in the second.",
 "distractors": [{"wrong": "Nothing - it was within 60 days",
                  "why_tempting": "the deadline is the more familiar rule",
                  "why_wrong": "the amount triggers a separate requirement"}]}

-- level 3 ------------------------------------------------------------------
{"question": "An employee drives 40 km from home to a client site, then 12 km
              back to the office where they normally work. How many kilometres
              are reimbursable at the standard rate?",
 "answer": "52",
 "level": 3,
 "support": ["reimbursed at the standard rate of 0.23 EUR per kilometre for
              journeys made in a private vehicle",
             "The standard rate does not apply to journeys between an
              employee's home and their normal place of work"],
 "step": "Neither leg is home-to-normal-workplace: the first ends at a client
          site and the second starts there. The exclusion does not apply, so
          both legs count.",
 "distractors": [{"wrong": "12",
                  "why_tempting": "the first leg starts at home, which looks
                                   like the excluded case",
                  "why_wrong": "the exclusion requires both ends to be home
                                and the normal workplace"}]}

The level 3 question is the one worth studying, because it shows what the definition buys. It is not longer or more convoluted than the others — it is shorter than the level 2 stem. What makes it level 3 is that the answer requires applying the exclusion to a journey the source never mentions, and deciding it does not apply. The step field states that reasoning, so a reviewer can check the level rather than trust it.

Its distractor is also the sharpest, and that is not a coincidence: the plausible wrong answer at level 3 is the one produced by applying the rule carelessly, which is exactly what the question is testing. When a level 3 distractor is hard to write, the question is usually really a level 2.

All three survive the no-source control — nothing here is general knowledge, because the rate, the deadline and the threshold are this organisation’s. That is why internal policy documents make good sources and public technical documentation often does not.

When it stops working

• The control discards more questions than it used to. The generator has drifted towards general knowledge. Usually the “must not be answerable without the source” clause has moved down the prompt or been shortened.
• level_3 questions have adjacent supporting spans. Check this mechanically — you have the spans and the source, so adjacency is computable. Adjacent spans at level 3 mean the level labels have become decorative.
• partially_correct dominates the grade distribution. Nearly always a grader hedging rather than a genuine middle. The requirement that it name both a present and a missing element is what prevents it; verify those arrays are actually populated.
• Distractors become obviously wrong. A sign the questions have got easier. Sample five and try to answer them without the source yourself.