Nested and Recursive Schemas: The Depth Limit Nobody Documents

There are two ceilings on schema depth and they behave completely differently. One is published, enforced at request time and easy to discover. The other is not a limit at all — it is a gradual change in what a failure looks like, and it has no error message.

Two different limits

The hard limit is a documented cap in the provider implementation. OpenAI’s Structured Outputs guide has published caps on nesting depth, on total object properties across a schema, and on the combined character length of names and enum values. Those numbers exist, they are enforced with a 400, and they have been raised at least once since the feature launched in August 2024 — so any figure you read anywhere, including here, needs checking against the current docs before you design around it. That is why the section below is a probe rather than a table.

The soft limit is the depth at which the model starts putting correct values in the wrong place. There is no announcement. Every response still validates. It is not a property of the provider at all; it is a property of the model, your schema and your document together, which is why nobody can publish a number for it that would mean anything for your case.

Why depth fails quietly

This is the important idea in the page. Constrained decoding does not remove errors, it relocates them — and nesting is where the relocation is most visible.

Without enforcement, a model that loses track of nesting emits unbalanced braces and your parser raises. Loud, immediate, traceable. With enforcement, unbalanced braces are unreachable: the mask will not emit } where the grammar does not allow one. So the model, having drifted, produces the only thing still available to it — a syntactically perfect document with the value in the wrong sub-object.

unconstrained, drifted:            constrained, drifted:

{ "parties": [                     { "parties": [
    { "name": "Acme",                  { "name": "Acme",
      "address": {                       "address": {
        "city": "Utrecht"                  "city": "Utrecht",
  }                                        "postcode": "1011AB" },   <-- belongs
                                         "role": "buyer" } ],            to party 2
  ^ JSONDecodeError                  "effective_date": "2026-01-04" }

  you find out in 3ms                ^ validates. you find out in Q3.

Two consequences follow directly. First, schema validation is not a quality signal for nested output; a 100% validation rate tells you nothing about placement. Second, the invariants that catch this are cross-field ones you have to write yourself — the count of parties, a total that must equal a sum of children, an id that must appear exactly once. Those are the properties worth testing.

A probe for the hard limit

Rather than trusting a number in any document, generate schemas of increasing depth and find where your endpoint stops accepting them. Runs in a few seconds, costs a handful of tokens, and is valid for the model and endpoint you actually use:

import os
from openai import OpenAI, BadRequestError

client = OpenAI(base_url=os.environ.get("BASE_URL"), api_key=os.environ["API_KEY"])
MODEL = os.environ["MODEL"]

def nest(depth: int) -> dict:
    """An object nested to the given depth, with a string leaf at the bottom."""
    node = {"type": "string"}
    for i in range(depth):
        node = {
            "type": "object",
            "additionalProperties": False,
            "required": [f"level_{depth - i}"],
            "properties": {f"level_{depth - i}": node},
        }
    return node

def accepted(depth: int) -> tuple[bool, str]:
    try:
        client.chat.completions.create(
            model=MODEL, max_tokens=64,
            messages=[{"role": "user", "content": "Fill every level with the word deep."}],
            response_format={"type": "json_schema", "json_schema": {
                "name": "probe", "strict": True, "schema": nest(depth)}},
        )
        return True, ""
    except BadRequestError as e:
        return False, str(e)[:220]

lo, hi, last_error = 1, 64, ""
ok, err = accepted(hi)
if ok:
    print(f"depth {hi} accepted -- no limit found below {hi}")
else:
    while lo < hi:                            # bisect the first rejected depth
        mid = (lo + hi + 1) // 2
        ok, err = accepted(mid)
        if ok:
            lo = mid
        else:
            hi, last_error = mid - 1, err
    print(f"deepest accepted depth: {lo}")
    print(f"first rejection said: {last_error}")

Read the rejection text, not just the depth. It usually names which cap you hit — nesting, property count or character budget — and those three interact: a wide schema hits the property cap long before the nesting cap, so the depth this probe reports for a one-property-per-level schema is an upper bound on what your real schema will get.

Run the same probe against every provider you route to. A schema that works on one model and 400s on another is a routine cause of intermittent failures in multi-provider setups, and the answer is usually to design to the tightest limit in your set rather than to branch.

Recursive schemas

Trees, comment threads, nested clauses, org charts. JSON Schema expresses these with a self-reference — "$ref": "#" for root recursion, or a named $defs entry that refers to itself. Hosted strict modes have supported recursive schemas in some form since fairly early, with the same caveat about checking current docs.

The failure mode is specific and worth anticipating: a grammar can always continue descending, so a model that starts producing nested nodes has no structural pressure to stop, and the generation runs into max_tokens. You get finish_reason: "length" and a truncated document. Bound it in the schema where you can — a depth integer field with a small enum of allowed values, or an explicit non-recursive leaf variant in an anyOf — and always check finish_reason before parsing.

Flattening, and when not to

The reliable answer to both limits is to stop nesting and emit a node list with parent references:

{ "nodes": [
    { "id": "n1", "parent_id": null, "kind": "clause", "text": "..." },
    { "id": "n2", "parent_id": "n1", "kind": "subclause", "text": "..." },
    { "id": "n3", "parent_id": "n1", "kind": "subclause", "text": "..." }
] }

Depth becomes one level regardless of how deep the tree is, so both caps stop mattering. Truncation degrades gracefully: a cut-off list is a shorter list rather than a broken tree. And errors become checkable — a dangling parent_id, a cycle, or a duplicate id are all assertions you can write in ten lines, where “this object is nested under the wrong parent” is not.

The cost is real, so do not do this reflexively. The model must now invent and track identifiers, which is a genuine burden on it, and you must rebuild the tree in code and handle the malformed cases. For two or three levels, nest. Flatten when the depth is unbounded, when the data is recursive by nature, or when the probe above tells you your natural schema does not fit.