Arrays and Counts: Why Models Return Seven of Ten Items

You ask for every line item on the invoice. There are ten. You get seven, the JSON validates, and nothing anywhere reports a problem. This is the single most reported structured-output bug and at least half the time the model is not the cause.

Four causes, wildly different fixes

Diagnose in that order, because the first three are cheap to rule out and the fourth is the expensive one to work on. Log finish_reason, usage.completion_tokens and the length of the text you sent on every extraction call and the first two answer themselves.

The constraint you thought you set

minItems and maxItems sit outside the documented supported keyword set for hosted strict modes. Depending on the stack, sending them either gets you a 400 naming the keyword — fine, you learn immediately — or a 200 where the keyword was quietly discarded.

The second case is the trap, because your schema is now a comment. You believe a floor is enforced, the API returned success, and the array is short. Nothing in any log says the constraint was never applied. Find out which of the two your endpoint does before you rely on it — and either way, put “every item, do not summarise or skip” in the array’s description, since that reaches the model whether or not the keyword survives.

Why counting is hard for a decoder

There is no counter. Each token is produced from the context, and the context contains the items already emitted — so “have I got them all” is not a lookup, it is a judgement the model re-makes at every array element from what it can see. Two structural consequences:

• Similar items compete. Six near-identical rows differing only in a number are the hardest case, because the emitted prefix looks like the remaining work. This is why tabular data with repeated structure is where the bug concentrates.
• Stopping is a token like any other. Closing the array is a decision competing with continuing it. Anything that makes the document feel finished — a summary line, a total, a page break — raises the probability of the closing bracket.

The corollary for schema design: never ask for the count before the array. A total_items field emitted first forces a commitment the model then has to satisfy, and it will produce a number and an array that disagree. Emitted after the array, the same field costs nothing and gives you a free internal consistency check — len(items) != total_items is a genuine signal that the model knew it had lost track.

Enumerate the input

The strongest fix converts an open-ended recall problem into a per-item decision. Number the candidate rows in the input, and require the output to carry the number back:

user message:
  Extract every line item. The document lines are numbered.
  Return one object per line that is a line item, carrying its line number.
  Do not skip lines. If a line is not a line item, omit it.

  [L001] Widget, large            2   12.50   25.00
  [L002] Bolt M6                 40    0.15    6.00
  [L003] --- subtotal ---                     31.00
  [L004] Delivery                 1    4.95    4.95

schema:  items[]: { source_line: string, description: string, qty: number,
                    unit_price: number, line_total: number }

Three things change at once. The model is now deciding “is L003 a line item?” four times, which is far easier than “have I found them all?” once. The output is checkable against the input without any ground truth, because you know which line numbers existed. And the exclusions become visible: a subtotal row correctly omitted looks identical to a row missed, until the line numbers tell you which happened.

The reconciliation function

def reconcile(sent_lines: list[str], items: list[dict]) -> dict:
    """sent_lines: the [Lnnn] labels you put in the prompt, in order."""
    expected = set(sent_lines)
    got = [i["source_line"] for i in items]

    return {
        "missing":     [l for l in sent_lines if l not in set(got)],  # unexplained
        "hallucinated": [g for g in got if g not in expected],        # not sent
        "duplicated":  [g for g in set(got) if got.count(g) > 1],
        "coverage":    round(len(set(got) & expected) / max(len(expected), 1), 3),
    }

# Then, in the pipeline:
r = reconcile(sent, items)
if r["hallucinated"]:
    fail(record, "cited lines that were not in the input")   # never retry into this
if r["coverage"] < 0.80:
    retry_smaller_chunks(record)                             # a chunking problem
# "missing" is expected and informative: those are the lines the model
# judged not to be items. Sample ten of them by hand once a week.

Note what missing is not. It is not an error list — a subtotal line should be missing. It is a review queue, and it is the only view you have of the model’s exclusion decisions, which are otherwise completely invisible. A model quietly dropping every discount row shows up here and nowhere else.

If enumeration is impossible because the input has no natural rows, the fallback is two passes: one call that lists identifiers only — short output, low truncation risk — and one call per identifier to extract its fields. It costs more calls and it is dramatically more reliable on long documents, because neither pass has to do recall and extraction at the same time.

Two smaller effects round this out. Order is not guaranteed and you should not depend on it: a model asked for items in document order will usually oblige, but nothing enforces it, so if order matters, extract the ordering key and sort in code. And duplicates behave the opposite way to omissions — uniqueItems sits outside the supported keyword sets alongside minItems, so a repeated item is not prevented either. Overlapping chunks make duplication the expected outcome rather than an anomaly, which is why the deduplication key belongs in your merge step and not in the schema.

Finally, resist the instinct to fix a short array by raising the temperature or by adding “be thorough” to the prompt. Neither addresses any of the four causes; the first trades a missing item for an invented one, and the second is the kind of change that appears to work on the three documents you test it on. Instrument the four causes, find out which one you have, and fix that.