Designing a Schema for a Document Type You Only See a Few Times a Year

Every extraction backlog contains a long tail: document types that turn up four, eight, twenty times a year. The instinct is to build each one a schema. The cost argument against that is real but modest. The argument that should actually settle it is that you will never be able to tell whether the schema still works.

The temptation, and the usual argument against it

A rare document type looks like an afternoon of work. It has ten fields. The fields are obvious. Somebody writes a schema, a prompt and a validation rule, checks it against the three examples on hand, and it works. This is genuinely satisfying and it is why the long tail of any mature extraction system contains dozens of these.

The usual objection is that the build does not pay for itself, and it is worth working that out honestly because it is the objection people expect and it is only half true.

The break-even, worked

Every input below is an assumption stated as one. None is a measurement, and the point of writing them out is that a reader with different numbers can redo the arithmetic in two minutes.

• Assume a fully-loaded engineering cost of $95 per hour.
• Assume nine hours to build a per-type extraction: schema, prompt, three labelled samples, one validation rule and a mapping into the review queue.
• Assume two hours per year of maintenance — a form revision, a provider change, a field that turned out to be optional.
• Assume manual keying of one such document takes 25 minutes at a loaded clerical rate of $32 per hour, so $13.33 per document.

build cost            = 9 x 95            =   855.00
maintenance per year  = 2 x 95            =   190.00
manual keying         =                       13.33 per document

first-year break-even = 855.00 / 13.33    =    64.1 documents

three-year horizon:
  bespoke total       = 855 + 2 x 190     = 1,235.00
  manual total        = 3 x V x 13.33     =    39.99 x V
  break-even          = 1,235 / 39.99     =    30.9 documents per year

So on these assumptions a type seen sixty-five times a year pays back within twelve months, and a type seen thirty-one times a year pays back over three. Below that, the bespoke schema is a loss on cost grounds alone. The arithmetic is easy and the conclusion is unsurprising.

It is also the weaker half of the argument, and it is worth saying so rather than resting on it. The assumptions are soft — keying time varies by a factor of three across document types, and the nine-hour build estimate is exactly the sort of number that is optimistic in proportion to how confident the person quoting it is. Somebody who disagrees with the inputs can move the break-even to anywhere they like, and both of you will be arguing about the wrong thing.

The real reason, which is not cost

At eight documents a year, you cannot build a test set.

That sentence is the argument. Every other quality mechanism in an extraction system rests on having a held-out set of documents with known-correct field values. A golden set of forty gives you a meaningful signal when a prompt changes, a model updates, or the form revises. A golden set of three, which is your entire corpus, gives you nothing: you cannot hold any of it out, because holding out one document removes a third of your evidence, and you tuned the prompt against all three, so passing on all three is a tautology rather than a result.

The consequences compound in a specific order, and they are worth spelling out because each one is invisible on its own:

• The provider changes the model under a stable name. High-volume types show a shift in error rate within days. A type with eight annual sightings shows nothing, because there is no denominator. The library already covers what a silent model update does to a pipeline; the point here is that the rare type is the one where it is undetectable.
• The form revises between sightings. Twelve months can pass between two instances of a rare type. The version you built against may have been withdrawn in between, and the first document of the new version arrives with no alarm attached to it.
• Nobody remembers the schema exists. A path exercised eight times a year is a path no engineer has read in months. When it does fail, the failure is diagnosed from scratch, and the diagnosis costs more than the nine hours saved.
• The failure is silent by construction. An extraction that returns plausible values for the wrong fields does not raise an error. With volume, the aggregate catches it. Without volume, the only detector is a human reading the output — which is precisely the work the schema was built to eliminate.

So the correct comparison is not bespoke-schema versus manual-keying. It is bespoke-schema-that-nobody-can-verify versus general-capture-plus-a-human. The second one is honest about where the judgement is, and the first one hides it behind a green tick.

What a general capture schema looks like

The alternative is one schema that captures what every business document has, with no per-type modelling at all:

{
  "document_kind_guess": "string, free text, not an enum",
  "parties": [ { "role": "issuer | recipient | other", "name": "...", "address": "..." } ],
  "dates":   [ { "label_as_printed": "Invoice Date", "value": "2026-04-02" } ],
  "identifiers": [ { "label_as_printed": "Claim No.", "value": "..." } ],
  "amounts": [ { "label_as_printed": "Total Due", "value": "1240.00", "currency": "USD" } ],
  "line_items": [ { "description": "...", "quantity": "...", "amount": "..." } ],
  "full_text": "...",
  "page_count": 3
}

Nothing in that is specific to any document type, which is exactly its virtue: it costs nothing to apply to a type you have never seen, and it is exercised by every document flowing through the system, so it is continuously tested by the high-volume traffic. The label-plus-value pairing is the load-bearing idea — it captures what the document printed without committing to what the field means, and that uncommitted form is what makes it survive a type it was not designed for. A rare document lands in it, a human reads six fields off a screen instead of hunting them across three pages, and nothing was built.

Where per-type knowledge does belong is in configuration rather than code: a mapping saying that for this document kind, the amount labelled “Total Due” is the payable and the date labelled “Due Date” is the deadline. Configuration can be added by whoever handles the document, carries no deployment, and when it is wrong it is wrong visibly. Compare the trade-offs in choosing between a nested and a flat schema and in designing before you have seen every variant; the position argued here is the far end of that spectrum, and it is the right end for the tail.

When to promote a type

The general schema is a default, not a doctrine. Promote a type to a bespoke schema when both of these are true, and not when only the first is:

1. The volume clears the break-even on your own numbers over the horizon you actually plan on. On the assumptions above that was around thirty-one documents a year over three years, or sixty-five over one. Redo it with your figures; the method is the transferable part, not the answer.
2. You can hold out a golden set without starving production — roughly, you see the type often enough that ten to fifteen documents can be labelled and set aside, and new instances arrive often enough that a regression would be caught within a sensible window. A monthly sighting is about the floor for that.
3. Then build it, and build the golden set first. The order matters: a schema written before the test set exists is a schema fitted to whichever documents happened to be on the desk that day. Writing the regression tests before you touch the prompt is the same discipline applied to a change rather than to a birth.

The two conditions land in a similar range on these assumptions, which is a coincidence and should not be read as confirmation of either. Notice that the second condition is a statement about observability rather than about money, and that it is the one that does not move when somebody argues with your hourly rate. A type you cannot test is a type you should not automate, at any cost per document.