LLM-as-Judge Prompt Generator

A weighted rubric in; a judge prompt out in both orderings, with the rule for combining them and the cost of judging a whole set.

What the assistant being graded was asked to do

CriteriaOne per line: name : weight : definition. The definition is the part that decides whether two runs of this judge agree.

Judgement form

Scale (pointwise)

Make the judge write its reasoning before its scoreA reference answer is supplied, and the prompt says it is one acceptable answer rather than the only oneAllow ties

Cases you will judge

cases

Repeats per case

runs

Judge input rateYour own rate for the judge model.

$ / 1M in

Judge output rate

$ / 1M out

Judge output tokens per callReasoning plus JSON. Reasoning-first prompts are longer.

tokens

Characters per tokenCharacters per token. An estimate — nothing here runs a tokeniser.

chars

Cost of judging one full set

$0.44

80 judge call(s): 40 case(s) × 1 repeat(s) × 2 for the swapped order. The prompt below is not counted against your task's own token budget — a judge is a second model call on top of it.

Criteria: 4
Weights sum to: 7.00
Heaviest criterion: Faithfulness (42.9%)
Judge prompt characters: 2,969
Estimated judge prompt tokens: ≈ 742
Judge calls per case: 2
Judge calls in total: 80
Cost per case: $0.01
Cost of judging this set daily for a month: $13.26

Pairwise judge prompt — run 1, answers in their given orderRead-only. Select all and copy.

You are grading the output of an assistant whose job is:

Answer a customer's question about their order, using only the order data supplied with the question.

You are not doing that job. You are grading how well it was done.

Criteria, in the order you must consider them:
- Faithfulness (weight 3, 42.9% of the score): Every claim in the answer is supported by the provided order data. An answer that invents a tracking number scores 1 however well it reads.
- Completeness (weight 2, 28.6% of the score): The answer addresses everything the customer asked, including the part they asked second.
- Tone (weight 1, 14.3% of the score): Plain, direct, no apology padding. Not scored on friendliness.
- Brevity (weight 1, 14.3% of the score): No sentence that does not carry information. Length is only a fault when it costs clarity.

You are given a reference answer. It is one acceptable answer, not the only one: an answer that differs from it but satisfies the criteria scores just as well. Do not reward similarity to the reference for its own sake.
For each criterion, write one sentence of reasoning BEFORE giving its score. Do not give any score until you have written its reasoning.
Judge only what is in front of you. Do not reward length, confidence, formatting or politeness unless a criterion above says to.

You are comparing two answers to the same input. They are labelled A and B. The
labels carry no information: they were assigned by a coin flip, and A is as
likely to be the worse answer as the better one.

If neither answer is better on a criterion, say “tie”. A tie is a real answer and is expected to be common.

<task_input>
{{input}}
</task_input>

<reference_answer>
{{reference}}
</reference_answer>

<answer_A>
{{answer_1}}
</answer_A>

<answer_B>
{{answer_2}}
</answer_B>

Reply with JSON only, matching this shape:
{
  "type": "object",
  "properties": {
    "reasoning": {
      "type": "string"
    },
    "per_criterion": {
      "type": "object",
      "properties": {
        "faithfulness": {
          "type": "string",
          "enum": [
            "A",
            "B",
            "tie"
          ]
        },
        "completeness": {
          "type": "string",
          "enum": [
            "A",
            "B",
            "tie"
          ]
        },
        "tone": {
          "type": "string",
          "enum": [
            "A",
            "B",
            "tie"
          ]
        },
        "brevity": {
          "type": "string",
          "enum": [
            "A",
            "B",
            "tie"
          ]
        }
      },
      "required": [
        "faithfulness",
        "completeness",
        "tone",
        "brevity"
      ],
      "additionalProperties": false
    },
    "winner": {
      "type": "string",
      "enum": [
        "A",
        "B",
        "tie"
      ]
    }
  },
  "required": [
    "reasoning",
    "per_criterion",
    "winner"
  ],
  "additionalProperties": false
}

Pairwise judge prompt — run 2, the same pair with the order swappedRead-only. This is the debiasing: identical prompt, A and B exchanged.

You are grading the output of an assistant whose job is:

Answer a customer's question about their order, using only the order data supplied with the question.

You are not doing that job. You are grading how well it was done.

If neither answer is better on a criterion, say “tie”. A tie is a real answer and is expected to be common.

<task_input>
{{input}}
</task_input>

<reference_answer>
{{reference}}
</reference_answer>

<answer_A>
{{answer_2}}
</answer_A>

<answer_B>
{{answer_1}}
</answer_B>

How to combine the two runsRead-only. Select all and copy.

# Aggregating two runs into one verdict.
#
# Position bias is the tendency of a judge to favour whichever answer it saw
# first, and it is large enough to swamp small real differences. The fix is not
# a prompt instruction — telling a judge to ignore order does not work — it is
# to ask twice with the order swapped and to treat disagreement as information.
#
#   run 1: answer_1 shown as A, answer_2 shown as B
#   run 2: answer_2 shown as A, answer_1 shown as B   <- the swapped prompt
#
# Then, per criterion and for the overall winner:

def verdict(run1: str, run2: str) -> str:
    """run1/run2 are "A", "B" or "tie" as the judge returned them."""
    first  = {"A": "answer_1", "B": "answer_2", "tie": "tie"}[run1]
    second = {"A": "answer_2", "B": "answer_1", "tie": "tie"}[run2]   # swapped
    if first == second:
        return first          # the judge agreed with itself: trust it
    return "tie"              # it flipped when the order flipped: no signal

# The share of pairs that come back "tie" BECAUSE the judge flipped is your
# measurement of this judge's position bias on this task. Report it. A judge
# that flips on 40% of pairs is not measuring quality, and a leaderboard built
# on it is measuring which answer got printed first.

# Weighted overall score, for the pointwise form:
#   0.429 × faithfulness + 0.286 × completeness + 0.143 × tone + 0.143 × brevity

What this assumes: the cost figures use YOUR rates and estimate the prompt at characters ÷ 4; they do not include the answers being judged, which are substituted into the {{answer}} placeholders and vary with your data — add their tokens to the input side. Position debiasing here means emitting both orderings and an aggregation rule that discards pairs where the judge flipped; that is a method for detecting and neutralising the bias, not a claim that this prompt is unbiased. Nothing on this page has been measured against human ratings — only you can do that, on your task. Everything on this page runs in your browser. Nothing you paste or open is uploaded, logged or sent anywhere.

A judge model is a measuring instrument, and the first thing to do with a measuring instrument is check that it gives the same reading twice. The best-documented way it does not is position: show the same pair of answers in the opposite order and a judge will often change its mind. If you never swap, that bias is silently baked into every comparison you run, and the answer that happened to be generated first wins.

Why the swapped prompt is the whole tool

Asking the judge to ignore the order does not work; the instruction is read and the preference persists. Asking twice does work, because disagreement between the two runs is itself the measurement. Pairs where the judge agrees with itself are signal. Pairs where it flips are noise, and the proportion that flip is a number you should be reporting alongside your win rate — a judge that flips on two pairs in five is not ranking anything.

The definitions do more work than the scale

Most judge inconsistency comes from criteria that were named but not defined. “Helpfulness, 1 to 5” means five different things across five calls. A definition that says what a 1 looks like, and names something the criterion explicitly does not cover, cuts the variance more than any change to the number of points on the scale. The weights matter too, and they are printed as percentages above so that “3, 2, 1, 1” is visible as the 43% / 29% / 14% / 14% split it actually is.

LLM-as-Judge Prompt Generator

Why the swapped prompt is the whole tool

The definitions do more work than the scale

The reasoning behind this

Related tools