Classification as Structured Output

Classification is the one structured-output task where JSON is overhead. You want one value from a closed set — which is one token, one forward pass, and a probability distribution you can actually use.

What a JSON classification wastes

The conventional approach asks for {"label":"refund_request"} under a strict schema. It works. It also generates a dozen or so output tokens where one would do, most of them structural, and it throws away the only quantitative thing in the system: the distribution over the label the model was choosing from.

On latency the difference is not subtle, because output tokens are sequential. A dozen output tokens is a dozen forward passes. One token is one, and for a classifier running per message on a support queue that is the difference between a service you can put in a synchronous path and one you cannot.

The cost side is less dramatic than it first appears and worth being honest about. Classification prompts are usually input-dominated — the message being classified plus the label definitions — so cutting twelve output tokens to one changes the bill by a few percent, not by an order of magnitude. Latency is the real prize, and the distribution is the prize nobody expects: a number you can threshold on, which is what turns a classifier into a cascade.

The single-token trick

Present the labels as a numbered list, ask for the number, cap the output at one token, and read the logprobs:

system: Classify the message. Answer with a single digit and nothing else.
        1 = refund_request   customer wants money back for a completed order
        2 = shipping_delay   customer asks where an order is
        3 = product_defect   item arrived broken or does not work
        4 = account_access   cannot log in, password, 2FA
        5 = other            none of the above

user:   <the message>

params: max_tokens=1, temperature=0, logprobs=true, top_logprobs=20

The response content is one character. The interesting part is logprobs.content[0].top_logprobs, which gives the top candidates at that position with their log-probabilities. Exponentiate the ones that are your labels, renormalise so they sum to one, and you have a posterior over your classes conditioned on the model having answered with a label at all — which is exactly the object you wanted and cannot get from a JSON response.

The classifier

import math, os
from openai import OpenAI

client = OpenAI(base_url=os.environ.get("BASE_URL"), api_key=os.environ["API_KEY"])
MODEL = os.environ["MODEL"]

LABELS = {"1": "refund_request", "2": "shipping_delay", "3": "product_defect",
          "4": "account_access", "5": "other"}

SYSTEM = ("Classify the message. Answer with a single digit and nothing else.\n"
          "1 = refund_request   customer wants money back for a completed order\n"
          "2 = shipping_delay   customer asks where an order is\n"
          "3 = product_defect   item arrived broken or does not work\n"
          "4 = account_access   cannot log in, password, 2FA\n"
          "5 = other            none of the above")

def classify(text: str) -> dict:
    r = client.chat.completions.create(
        model=MODEL, max_tokens=1, temperature=0,
        logprobs=True, top_logprobs=20,
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": text}],
    )
    top = r.choices[0].logprobs.content[0].top_logprobs

    # Sum probability mass per label: " 1" and "1" are different tokens
    # and both mean label 1.
    mass = {name: 0.0 for name in LABELS.values()}
    seen = 0.0
    for cand in top:
        key = cand.token.strip()
        if key in LABELS:
            p = math.exp(cand.logprob)
            mass[LABELS[key]] += p
            seen += p

    if seen == 0.0:                      # no label in the top-k at all
        raise OffDistribution(r.choices[0].message.content)

    probs = {k: v / seen for k, v in mass.items()}          # renormalise
    ranked = sorted(probs.items(), key=lambda kv: -kv[1])
    return {
        "label":  ranked[0][0],
        "p":      round(ranked[0][1], 4),
        "margin": round(ranked[0][1] - ranked[1][1], 4),    # the useful number
        "probs":  probs,
        "unseen_mass": round(1 - seen, 4),   # mass on non-label tokens
    }

class OffDistribution(Exception): pass

margin is the number to build on. The top probability alone conflates “confident” with “only one plausible option”; the gap to the runner-up is what tells you whether the model was actually deciding between two things. Route low-margin cases to a bigger model or to a human and you have a cascade whose cost you control by moving one threshold.

unseen_mass is the health check. It is the probability the model put on tokens that were not labels at all. Small is normal; large means the model wanted to say something else, which usually means your label set does not cover the input.

Because the output is a distribution rather than a word, this classifier is evaluable in the ordinary way. Score a few hundred labelled examples, build the confusion matrix, and read it rather than the headline accuracy: almost always one or two off-diagonal cells dominate, and they name a pair of labels whose definitions overlap. Fixing that is a prompt edit, not a model change. Sweep the margin threshold over the same set and you get the trade-off curve between what you escalate and what you get wrong, which is the number to bring to whoever owns the cost of a mistake.

The traps

• Leading whitespace. Most BPE tokenisers treat "1" and " 1" as different tokens. The .strip() above is not tidying, it is the correctness fix, and omitting it is why implementations of this pattern silently return zeros.
• Multi-token labels break it entirely. refund_request is several tokens; there is no single position whose distribution is over your classes. Digits and single capital letters are safe. Beyond about twenty classes, use two-stage classification rather than reaching for longer labels.
• top_logprobs is truncated. You get the top k candidates, not the whole vocabulary. A label outside the top k gets probability zero in this scheme rather than a small value. With five labels and k = 20 this is rarely binding; with fifteen labels it is.
• Not every endpoint returns logprobs. Support varies by provider and by model, and some return the field as null without erroring. Check before you depend on it.
• Label order and label names both matter. They are in the prompt, so they are evidence. Do not assume the digits are neutral — if 5 = other systematically absorbs a class, try reordering before you conclude anything about the model.

When to use the JSON version instead

The single-token classifier gives up two things. It cannot return multiple labels, and it cannot return anything alongside the label — no extracted span, no reason, no secondary field. If you need any of that, you need a JSON response and the extra tokens are the price.

It also gives up the reasoning room that a leading reasoning field would provide, which matters for genuinely hard judgements. The rule of thumb: single-token for high-volume routing where the decision is a surface judgement, JSON with a reasoning field for low-volume decisions where being right is worth ten extra tokens. Nothing stops you running both — cheap classifier first, escalate on low margin.