Classification Prompts With Stable Labels

A classification prompt is mostly its label set. Get the definitions, the boundaries between them and the escape hatch right, and the instruction wrapped around them barely matters. Here is the whole thing, for support ticket triage, with the reasoning for each part.

The prompt

Classify the message below into exactly one label from <labels>.

<labels>
billing          The customer is asking about an amount charged, an invoice, a
                 refund, a payment method, or the price of a plan.
bug              Something in the product does not do what it is documented to
                 do. Requires a claim that the behaviour is wrong, not merely
                 unwanted.
feature_request  Something the product does not do and is not documented to do.
how_to           The product can already do it and the customer does not know
                 how.
account_access   The customer cannot sign in, reset a password, or reach their
                 own data at all.
other            None of the definitions above describes this message.
</labels>

<tie_breaks>
Apply in order. Stop at the first one that applies.
1. billing beats every other label when a specific amount, invoice number or
   charge date is mentioned.
2. account_access beats every other label when the customer cannot reach the
   product at all.
3. bug beats feature_request when the customer cites documentation, a
   previously working behaviour, or an error message.
4. how_to beats bug when the customer reports confusion but no error and no
   changed behaviour.
5. If two labels still tie, choose the one that appears first in <labels>.
</tie_breaks>

Use "other" when no definition fits. "other" is a correct answer, not a
failure. Do not stretch a definition to avoid it.

Return only this JSON:
{"label": "<one label name>",
 "evidence": "<verbatim span from the message, at most 15 words>",
 "runner_up": "<the label you considered second, or null>",
 "tie_break_used": "<the rule number, or null>"}

If label is "other", "evidence" must be the span that made none of the
definitions fit.

<message>
{{message}}
</message>

Definitions, not label names

A bare list of names — billing, bug, feature_request, how_to — leaves every boundary to the model’s prior, which means the boundary moves when you change model, and moves again when you add a label. A definition pins it, and the useful definitions are the ones that state what is excluded, not what is included.

Compare the two halves of the bug definition. “Something in the product does not do what it is documented to do” is the inclusive half and it is nearly useless on its own, because most complaints can be read that way. “Requires a claim that the behaviour is wrong, not merely unwanted” is the half that does the work: it draws the line against feature_request, which is the only neighbour bug is ever confused with.

Write definitions in pairs like that, one per boundary you actually observe being crossed. You will not need one for every possible pair — in a six-label set there are fifteen pairs and typically three of them account for nearly all the disagreement. Find those three by looking at the runner_up field, which is why it is in the schema.

Tie-breaks as ordered pairs

The reason tie-breaks are written as ordered, numbered rules rather than as a priority list of labels is that priority lists are wrong. “billing > account_access > bug > how_to” asserts a total order over labels, and no real label set has one: billing should beat bug when there is an amount in the message and should lose to it when there is not.

A pairwise rule with a condition attached says exactly that. It also makes each rule independently testable — you can write three examples per rule and know which rule regressed when one of them flips.

Rule 5 is the one people leave out, and it is the one that makes the output stable. Without a final positional tie-break, two genuinely balanced labels get resolved by whatever the sampler does, so the same message classifies differently on Tuesday. With it, plus temperature 0, a tie has a defined answer. It is an arbitrary answer, and arbitrary is fine; unstable is not, because an unstable label makes your eval set noisy and hides real regressions.

The tie_break_used field turns the rules into telemetry. A rule that never fires is dead weight in the prompt and can be deleted. A rule that fires on 30% of messages is describing a boundary your definitions should have handled.

Blessing the none-of-these label

Two sentences do the work here: “other” is a correct answer, not a failure. and Do not stretch a definition to avoid it. Both are necessary and the second is the one that changes behaviour.

Instruction-tuned models are shaped to be helpful, and returning other reads as unhelpful. Left to itself, the model will find the nearest label and take it — which is the worst possible outcome, because a stretched label is indistinguishable from a correct one in your metrics. You lose the signal that your label set no longer covers your traffic, which is the single most useful thing a classifier can tell you.

The requirement that other still carries an evidence span is what makes it actionable rather than a bin. Fifty other decisions with fifty spans is a readable document, and it is usually obvious from it whether you have one missing label or fifty genuinely unrelated messages.

The evidence span

Requiring a verbatim span costs about twelve output tokens and buys three things:

• It is checkable. Assert it is a substring of the message. When that assertion starts failing, something changed — usually the model, sometimes your whitespace handling.
• It makes review fast. A reviewer reading label plus span can audit a hundred decisions in ten minutes. A reviewer reading label plus full message cannot.
• It constrains the decision. The model must locate something in the text that supports the label before committing to it, which is a weaker version of the grounding requirement that makes extraction prompts reliable.

Cap the span. Without a limit the model returns the whole message, which is true and useless. Fifteen words is enough to be evidence and short enough to scan in a column.

Three messages, worked

One message per interesting behaviour: a tie-break firing, an escape label being used correctly, and a case where the runner-up is the informative part.

-- 1: a tie-break decides it ------------------------------------------------
message: "I was charged 49 EUR on the 3rd but the export still crashes when I
          pick more than 500 rows. Invoice 88231."

{"label": "billing",
 "evidence": "I was charged 49 EUR on the 3rd",
 "runner_up": "bug",
 "tie_break_used": "1"}

Both definitions fit: there is a specific charge AND a documented feature that
crashes. Rule 1 decides it, and "runner_up": "bug" is what tells your queue
this ticket has a second problem in it. Without the tie-break, this message
classifies differently depending on which half the model read as the point.

-- 2: "other", used correctly -----------------------------------------------
message: "Are you hiring? I saw your engineering blog and wanted to ask about
          the platform team."

{"label": "other",
 "evidence": "Are you hiring?",
 "runner_up": null,
 "tie_break_used": null}

No definition fits and the model does not stretch one. Twenty of these in a
week is not a missing label; it is a contact form pointed at the wrong inbox.

-- 3: the runner-up is the finding ------------------------------------------
message: "The CSV export gives me commas inside the fields so Excel splits
          them wrong. Is there a way to change the delimiter?"

{"label": "how_to",
 "evidence": "Is there a way to change the delimiter?",
 "runner_up": "bug",
 "tie_break_used": "4"}

Rule 4 is right: no error, no changed behaviour, just an unquoted-CSV
complaint phrased as a question. But this pair appearing repeatedly is the
signal to read, because "how_to beats bug" is only correct while the export
genuinely is configurable.

The third example is the one to sit with. The classification is correct under the rules, and a rising count of exactly this pair means the rules are describing a product that has changed. That is the kind of thing a label alone can never tell you and a label plus runner-up tells you for twelve extra output tokens.

The drift signal, and what to change

Three numbers, tracked weekly, in order of how early they move:

What to change first, in order: the definition of the label whose boundary the runner_up data implicates; then a tie-break rule for that pair; then, only if the other spans genuinely justify it, a new label. Adding a label is the most expensive change because it invalidates your eval set and every threshold built on the old distribution — record that dependency in your recipe metadata so the next person knows.

Do not tune the prompt against production traffic alone. Keep a fixed set of a hundred or so labelled messages, run it on every change, and treat any movement on the pinned cases as a regression — the workflow in regression testing for prompts applies here almost unchanged.