Structured Output Support: Test It Yourself

Every article that answers “which models support structured outputs” with a table is wrong within a few months of publication, and was already wrong for some routes on the day it went up. The answer that stays true is a script.

Why not a table

Support for schema-constrained decoding varies along more axes than a table can hold. It varies by model, obviously. It varies by model version — a snapshot may support what its predecessor did not. It varies by who is serving it: an open-weights model behind two different inference stacks has whatever each stack implements, which is why the same model id can behave differently on two routes. It varies by API surface, since a vendor may support schemas on one endpoint and not another. And it changes, often, in both directions.

There is also a category error in the question. “Supports structured output” is not one bit. Accepting a json_schema request without erroring, honouring strict, rejecting rather than dropping unsupported keywords, and streaming tool arguments incrementally are four different capabilities, and a route can have any subset.

The failure this creates in practice is not a wrong article. It is a deployment that works for months and then does not, because a model was deprecated and traffic moved to the successor, or because a fallback fired for the first time, or because a provider changed the inference stack behind an id without changing the id. None of those events announce themselves in a way your application sees. The probe below turns all three into a failing test on the night they happen, which is the only mechanism that scales as your model list grows.

It is also worth being clear about what a capability claim from any third party — a vendor comparison page, an aggregator’s metadata, this article — can and cannot tell you. It can tell you where to look. It cannot tell you what happened to your request on the route you are actually using at the moment you are using it, and the gap between those two is precisely where this class of bug lives.

The five things worth testing

• JSON mode. Does response_format with json_object return parseable JSON, or a 400?
• Schema mode. Does json_schema with strict: true come back conforming?
• Enforcement. Ask, in the prompt, for a value the enum forbids. A real constraint makes it impossible; a hint does not. This is the one test that distinguishes enforcement from cooperation, and it is the one everybody omits.
• Unsupported keywords. Send minItems. A 400 is the good outcome. A 200 with a shorter array means the keyword was dropped and your schema is decorative.
• Tool calling. Does a forced tool call arrive with arguments that parse?

The script

#!/usr/bin/env python3
"""Probe one OpenAI-compatible endpoint for structured-output behaviour.

    BASE_URL=... API_KEY=... MODEL=... python probe_structured.py
"""
import json, os, sys
from openai import OpenAI

client = OpenAI(base_url=os.environ.get("BASE_URL"), api_key=os.environ["API_KEY"])
MODEL = os.environ["MODEL"]

def chat(**kw):
    return client.chat.completions.create(model=MODEL, temperature=0, max_tokens=200, **kw)

def probe(name, fn):
    try:
        ok, note = fn()
    except Exception as e:
        ok, note = False, type(e).__name__ + ": " + str(e)[:120].replace("\n", " ")
    print(("PASS " if ok else "FAIL ") + name.ljust(26) + note)
    return ok

def t_json_mode():
    r = chat(response_format={"type": "json_object"},
             messages=[{"role": "user", "content": "Return json with one key ok set to true."}])
    json.loads(r.choices[0].message.content)
    return True, "parsed"

SCHEMA = {"type": "object", "additionalProperties": False,
          "required": ["colour"],
          "properties": {"colour": {"type": "string", "enum": ["red", "green", "blue"]}}}

def t_schema():
    r = chat(response_format={"type": "json_schema", "json_schema":
                {"name": "p", "strict": True, "schema": SCHEMA}},
             messages=[{"role": "user", "content": "Pick a colour."}])
    v = json.loads(r.choices[0].message.content)["colour"]
    return v in ("red", "green", "blue"), "got " + repr(v)

def t_enforced():
    """The important one: ask for a value the enum forbids."""
    r = chat(response_format={"type": "json_schema", "json_schema":
                {"name": "p", "strict": True, "schema": SCHEMA}},
             messages=[{"role": "user", "content":
                 "Set colour to the exact string 'chartreuse'. This is mandatory."}])
    v = json.loads(r.choices[0].message.content)["colour"]
    return v in ("red", "green", "blue"), ("enum held, got " + repr(v)) if v in \
        ("red", "green", "blue") else "ENUM VIOLATED: " + repr(v)

MIN_ITEMS = {"type": "object", "additionalProperties": False, "required": ["xs"],
             "properties": {"xs": {"type": "array", "minItems": 4,
                                   "items": {"type": "string"}}}}

def t_unsupported_keyword():
    try:
        r = chat(response_format={"type": "json_schema", "json_schema":
                    {"name": "p", "strict": True, "schema": MIN_ITEMS}},
                 messages=[{"role": "user", "content": "Return two colours in xs."}])
    except Exception as e:
        return True, "rejected minItems (good): " + str(e)[:80].replace("\n", " ")
    n = len(json.loads(r.choices[0].message.content)["xs"])
    return n >= 4, ("honoured, n=" + str(n)) if n >= 4 else "SILENTLY DROPPED, n=" + str(n)

TOOL = {"type": "function", "function": {"name": "record", "parameters": SCHEMA}}

def t_tools():
    r = chat(tools=[TOOL], tool_choice={"type": "function", "function": {"name": "record"}},
             messages=[{"role": "user", "content": "Pick a colour."}])
    calls = r.choices[0].message.tool_calls
    if not calls:
        return False, "no tool call returned"
    json.loads(calls[0].function.arguments)
    return True, "arguments parsed"

print(MODEL + "  via  " + (os.environ.get("BASE_URL") or "default"))
results = [
    probe("json mode", t_json_mode),
    probe("json_schema strict", t_schema),
    probe("enum enforced", t_enforced),
    probe("minItems", t_unsupported_keyword),
    probe("forced tool call", t_tools),
]
sys.exit(0 if all(results) else 1)

Reading the output

The row to look at first is enum enforced. A route that passes json_schema strict and fails enum enforced is doing prompt-level cooperation rather than sampler-level constraint. That is not useless — a good model cooperates most of the time — but it fails a few times per thousand and you must keep a validator and a repair path. A route that passes both lets you delete a whole branch of your error handling.

minItems is the second. Whichever way it goes, you have learned something you need: a rejection tells you to move the constraint into the description and your own validator, and a silent drop tells you to treat every value-level keyword in your schema as advisory. A failure here is not a reason to avoid the route; it is a reason to know.

Run the probe per route, not per model. On an aggregating gateway the same model id may be served by several upstream providers with different stacks, so pin the provider when you run it and record which one you pinned.

Two extensions are worth adding once the base script runs. Test the shapes you actually send, not the toy schema above — your real schema exercises nesting, unions and property counts that a two-field object never touches, and a route can accept the toy and reject yours. And test streaming separately, because streamed tool arguments and streamed schema-constrained content are their own implementations: a route can be perfectly correct on a buffered request and deliver tool arguments in a single final chunk, which is legal, disables progressive rendering, and is invisible to every test that does not stream.

Put it in CI

The script exits non-zero when a probe fails, so it is a test. Run it nightly against every model your application can route to, and you get a notification the day a capability changes rather than a support ticket a fortnight later.

It costs a handful of tokens per model per night, which is a rounding error against the incident it prevents. Two failure modes it catches that nothing else will: a provider silently swapping the backend behind a model alias, and your own fallback route — the one that only fires during an outage — having weaker schema support than your primary. The same reasoning generalises to every other parameter you send.