Testing That a Model Never Returns a Value Outside an Enum

An enum field looks like the safest part of a structured output. It is the part most likely to arrive as a value you did not define, because the guarantee you are relying on may not be the one you think.

Who is actually enforcing the enum

There are three different mechanisms behind the phrase “the model returns one of these values”, and they give very different guarantees.

• Constrained decoding. The provider compiles your schema into a grammar and masks the sampler so tokens that would leave the grammar cannot be chosen. Under this mechanism an out-of-enum string is not improbable, it is unreachable — but it only applies when you have opted into the strict mode that enables it, with a schema that satisfies its restrictions.
• Instruction following. The enum is described in a prompt or a non-strict schema and the model is asked to comply. This usually works and occasionally does not, and the failures cluster on exactly the inputs where none of the members fit.
• Your validator. Whatever the provider does, the value crosses a boundary into your code and you parse it. This is the only layer you control, and it is the one the test is really about.

The practical consequence: write the test so it passes under constrained decoding for the right reason, and still passes under instruction following because the code handles the violation. The difference between the two is covered in JSON mode against structured outputs.

The documented limits

Strict schema support is not unbounded, and the boundaries are where the guarantee quietly stops applying. OpenAI’s structured outputs documentation states that a schema may have up to 1,000 enum values across all enum properties, and that where a single enum property has more than 250 string values, the total string length of all enum values cannot exceed 15,000 characters. The same guide requires every field to be marked required and every object to set additionalProperties to false — see OpenAI’s structured outputs guide for the current list of supported keywords.

Those numbers matter to a test because a schema that exceeds them is rejected or silently unenforced, and the schema that exceeds them is usually generated. A category enum built from a database table has 240 members today and 260 next quarter. Assert the invariant in a unit test that never calls a model:

import { describe, it, expect } from "vitest";
import { answerSchema } from "../schema";

const enums = collectEnums(answerSchema); // string[][]

describe("schema stays inside strict-mode limits", () => {
  it("has at most 1000 enum values in total", () => {
    expect(enums.flat().length).toBeLessThanOrEqual(1000);
  });
  it("respects the string-length rule above 250 values", () => {
    for (const values of enums) {
      if (values.length > 250) {
        expect(values.join("").length).toBeLessThanOrEqual(15_000);
      }
    }
  });
});

The adversarial case

A test that feeds ordinary inputs and asserts the value parses proves almost nothing, because ordinary inputs map cleanly onto a member. The case that produces a violation is the one where no member is right:

1. An input outside the taxonomy. A support classifier with the members billing, bug and feature, given a message about a data-deletion request.
2. An input that spans two members. A message that is genuinely both a bug and a billing issue. Models often answer with a concatenation like bug/billing, which is exactly the shape a loose parser accepts.
3. An empty or nonsense input. A single character, a blank string, a base64 blob.
4. An input containing an instruction. A message whose text says the category is URGENT_ESCALATE. This is the enum version of the injection tested in sub-agent output validation, and under instruction following it works more often than you would like.

import { z } from "zod";

const Category = z.enum(["billing", "bug", "feature", "other"]);

it.each(adversarialInputs)("stays inside the enum for %s", async (name, input) => {
  const raw = await classify(input);            // returns the parsed JSON object
  const result = Category.safeParse(raw.category);
  expect(result.success, `got ${JSON.stringify(raw.category)} for ${name}`).toBe(true);
});

Put the received value in the failure message. An enum failure whose message is “expected true, got false” costs you a debugging session; one that says the model answered bug/billing tells you the fix immediately.

Give it a legal way out

The fix for the first adversarial case is not a better prompt. It is another member, or a nullable field. A model forced to choose between three wrong answers will choose one, and a constrained decoder guarantees only that the answer is in the set — not that it is correct. Under constrained decoding, removing the escape hatch does not remove the uncertainty; it converts a visible violation into a confident misclassification, which is strictly harder to detect.

So the test suite grows an assertion in the opposite direction: on the out-of-taxonomy inputs, assert the value is other rather than merely in the enum. That is an assertion about behaviour, so run it over several samples and treat it as a rate, and keep it in the evaluation suite rather than in the unit tests. Watching the other rate in production is also the cheapest taxonomy-drift alarm you will ever build: a rate that climbs means the world has a category your enum does not.

Testing the rejection path

Finally, test what happens when validation fails, because that code also only runs on the days that matter. Bypass the model entirely and hand your validator the bad values directly.

it.each([
  ["unknown member", "urgent"],
  ["concatenated", "bug/billing"],
  ["case variant", "Billing"],
  ["whitespace", " bug "],
  ["null", null],
  ["number", 3],
])("rejects %s", (_n, value) => {
  expect(Category.safeParse(value).success).toBe(false);
});

The case and whitespace rows force a decision you would otherwise make by accident. If your product should accept Billing, normalise before validating and assert the normalisation; if it should not, assert the rejection. What you must not have is a validator that rejects it and a downstream that lowercases it later.

Then assert the consequence: an invalid category produces a typed failure the caller can branch on, is counted under a metric name, and does not become the string undefined in a database column. An enum violation that survives into storage is not a validation bug any more, it is a data-quality one, and it outlives the deploy that caused it.