Testing Strict Mode for Structured Outputs the Way You Can Actually Test It

You cannot meaningfully test that a provider’s constrained decoder works. You can test the three things around it that are yours, and those are where the bugs are.

What strict mode actually promises

OpenAI’s structured outputs guide describes a format object carrying type of json_schema, a name, the schema itself and a strict flag. On the Responses API it is nested under text.format:

{
  "model": "gpt-5.2",
  "input": [{ "role": "user", "content": "..." }],
  "text": {
    "format": {
      "type": "json_schema",
      "name": "invoice_extract",
      "strict": true,
      "schema": {
        "type": "object",
        "additionalProperties": false,
        "required": ["vendor", "total_cents", "currency"],
        "properties": {
          "vendor": { "type": "string" },
          "total_cents": { "type": "integer" },
          "currency": { "type": "string", "enum": ["GBP", "EUR", "USD"] }
        }
      }
    }
  }
}

The guide is explicit about what the schema must satisfy for strict to be accepted: additionalProperties set to false on every object, every field listed in required — there are no optional properties — and a root that is an object rather than an anyOf. It also documents ceilings: a maximum of 5,000 object properties across ten levels of nesting, up to 1,000 enum values in total, and a 120,000 character limit on all property names and enum values combined. See OpenAI’s structured outputs guide for the current list, which has grown before and will again.

The promise is about shape. It says the string you get back parses as JSON and conforms to the schema. It says nothing about whether total_cents is the right number.

Assertion one: the schema is rejected at request time

The most valuable test here fails before a single token is generated. Strict mode validates your schema up front and errors on unsupported constructs, so a schema that has drifted — somebody added an optional field, or a nested object without additionalProperties: false — is a request-time 400 in production and can be a request-time 400 in your test suite for the same reason.

You do not need the provider for this. Write a test that walks the schema you ship and asserts the invariants strict mode requires. It runs in milliseconds, it needs no credential, and it names the offending path in its failure message rather than making you read a provider error that points at the root.

import { describe, expect, it } from "vitest";
import { invoiceExtractSchema } from "../src/schemas";

function objects(node: unknown, path = "$"): [string, Record<string, unknown>][] {
  if (typeof node !== "object" || node === null) return [];
  const self = node as Record<string, unknown>;
  const here: [string, Record<string, unknown>][] =
    self.type === "object" ? [[path, self]] : [];
  const props = (self.properties ?? {}) as Record<string, unknown>;
  return Object.entries(props).reduce(
    (acc, [key, value]) => acc.concat(objects(value, path + "." + key)),
    here,
  );
}

describe("invoice schema is legal under strict mode", () => {
  for (const [path, node] of objects(invoiceExtractSchema)) {
    it("closes " + path, () => {
      expect(node.additionalProperties).toBe(false);
      expect(new Set(node.required as string[]))
        .toEqual(new Set(Object.keys(node.properties as object)));
    });
  }
});

Assertion two: re-validate with a validator you own

Parse the response and run it through an independent validator — Ajv, Pydantic, whatever your language offers — against the same schema object you sent. This is not distrust of the provider. It is that the schema you sent and the schema your downstream code expects are two things maintained in two places unless you make them one thing, and the re-validation is what makes them one thing.

It also covers the routes strict mode does not reach: a fallback to a model or provider that does not support the feature, a cached response recorded before the schema changed, and any path where a human edited a stored result. Assert on the parsed object’s conformance, never on the raw string — key order and whitespace are not part of the contract and a string comparison makes them part of your test.

Assertion three: the refusal branch

A model can decline. OpenAI’s guide documents refusals as a distinct content object in the output, with a type of refusal and the explanation in a refusal field, rather than as schema-conforming JSON. Code that reaches straight for the text part and parses it will throw on a refusal, and the exception surfaces as a JSON parse error, which sends whoever is on call looking for a malformed-output bug that does not exist.

Test it with a fixture, not with a live prompt engineered to be refused: put a refusal-shaped response in your mock, and assert your handler returns a typed refusal result rather than throwing. That test is deterministic and stays true regardless of what any model decides to decline this quarter.

The bug strict mode cannot catch

An enum with three currencies guarantees you get one of three currencies. It does not guarantee you get the right one, and because the output is always well-formed, every downstream layer waves it through. This is the failure mode that makes people over-trust strict mode: the class of error it eliminates is loud, and the class it leaves behind is silent.

So the eval that matters is a labelled set with expected field values, scored per field, and it is a different artefact from the tests above. Keep them separate: schema conformance is a unit test that runs on every commit with no model call at all, and field-level accuracy is a golden dataset scored on a schedule. Conflating them gives you a slow suite that cannot tell a broken schema from a worse model.

There is a design consequence hiding in the “no optional fields” rule that bites people about a month in. Because strict mode requires every property to appear in required, the way to express “this value may be absent” is a nullable type — a union of the type and null — not an omitted key. That changes what your downstream code must handle: the field is always present and sometimes null, rather than sometimes missing. If your consumer was written against a schema with genuinely optional keys, it will now see explicit nulls where it expected undefined, and a check like “does this key exist” becomes always true. Assert on the null case directly in the re-validation test, with a fixture that has every nullable field set to null, because it is the shape your extraction code will actually meet on a document that is missing half its data.

Finally, keep the schema in one module and import it into both the request builder and the tests. A schema that is defined inline in the request is a schema no test can walk, and the request-time check above quietly becomes a check of a copy. See how JSON mode differs from schema-constrained output if you are still deciding which of the two you are testing.

One operational note that saves an afternoon. The documented ceilings — property counts, nesting depth, total enum values, the combined character limit on property names and enum values — are easy to approach without noticing when a schema is generated from a type definition rather than written by hand. A generated schema over a large domain enum grows every time somebody adds a row to the source of truth, and the request that crosses the limit fails at request time for all users at once. Add an assertion that counts the properties, the depth and the total enum values in the generated schema and fails while the numbers are still comfortable rather than at the boundary. It is three lines beside the walk you already wrote, and it converts a production outage into a pull request comment.