Extracting Structured Fields From an Intake Form With Mixed Handwriting and Print

A page-level accuracy figure on an intake form is close to a lie. The preprinted labels, the instructions and the footer are typeset, will be recognised almost perfectly, and make up most of the characters on the page. All of the risk is in the small fraction that somebody wrote by hand.

Two rendering processes on one page

A clinic intake sheet, an insurance application, a school enrolment form, a warranty registration: the shared structure is a typeset skeleton with handwritten values dropped into it. The two layers came from completely different processes and they behave completely differently under recognition.

The typeset layer is a known quantity. It is the same on every copy of the form, it is machine-set at a consistent size, and — this is the part worth exploiting — you already know what it says. If you have the blank form, you have the labels, their positions and their order before any document arrives. That turns recognition of the labels from a task into a lookup, and turns the interesting problem into registration: aligning this scan to the known template so that the label positions transfer.

The handwritten layer is where every error lives. It varies per person, per pen, per mood, and it can be in block capitals, in cursive, or in a hybrid; recognition quality on cursive is materially worse than on hand-printed block letters and varies sharply between models, which is a routing decision as much as a prompting one. The library’s handwriting recognition page covers the recognition side; what follows is the part that is specific to this being a form.

Deriving the answer region from the label

The geometric relationship between a label and its answer is fixed by the form’s design and is one of a small number of patterns:

• Label left, answer right, bounded by the next rule or the next label’s left edge.
• Label above, answer on the line beneath, bounded below by the next label.
• Label above a comb — a row of boxed character cells. This is the easiest case in the whole document because segmentation is given to you: each cell holds one character, so recognition is a sequence of single-character problems rather than a word problem.
• Label beside a checkbox or a set of them, where the answer is a mark rather than characters.

Deriving the region and cropping to it before recognition does two things. It removes the typeset label from the input, so the model cannot answer with it — the failure where “Date of Birth” produces a plausible date from a blank field is largely a consequence of the label being in the crop. And it makes every quality number you compute afterwards a number about the answer.

Registration is what makes this survive real scans. A phone photograph of a form is rotated a degree or two, keystoned, and scaled unpredictably, so template coordinates do not transfer directly. Aligning on the form’s own printed features — the rules, the box corners, any fiducial marks the form designer provided — before applying template regions is the difference between a region-based approach working and it silently reading the neighbouring field.

Confidence over the answer, not the page

Once answers are cropped, the confidence question becomes answerable. The number you want for each field is a property of the recognition of that field’s answer region and nothing else. Concretely:

// Wrong: one number for the document, dominated by typeset text.
{ "document_confidence": 0.97, "fields": { "date_of_birth": "1974-03-11" } }

// Right: a number per field, computed over the answer region only,
// plus what it was computed from.
{
  "fields": [
    {
      "name": "date_of_birth",
      "value": "1974-03-11",
      "raw": "11/3/74",
      "region": { "page": 1, "bbox": [0.34, 0.21, 0.58, 0.25] },
      "medium": "handwritten",
      "confidence": 0.61,
      "validation": { "rule": "date", "passed": true, "ambiguity": "day_month_order" }
    },
    {
      "name": "form_version",
      "value": "INT-4 (rev 2025-06)",
      "medium": "printed",
      "confidence": 0.99,
      "validation": null
    }
  ]
}

Carrying medium per field is what stops the two populations being averaged again downstream by somebody computing a summary — and it is also what makes calibrating those numbers possible at all, since the two populations calibrate differently. A review queue built on these numbers behaves sensibly: it surfaces handwritten answers in the uncertain band and never surfaces a printed form version that was legible on every copy. The general design of that queue and how to present the numbers to a reviewer is covered by the library’s confidence UX page.

Constrained fields need no confidence

A large share of intake fields have a format, and where a format exists it beats a confidence score outright, because it is a check on the answer rather than a report of the model’s comfort.

A postal code either matches the pattern for its country or it does not. A phone number has a digit count. A date either resolves to a real calendar date or it does not, and a date of birth that resolves to next month is wrong regardless of how confident the recogniser was. An account or member number issued by your organisation can be checked against your own records — membership beats everything, and where the identifier carries a check digit, the arithmetic settles it outright.

The productive move is to run the constraint first and use confidence only where no constraint exists. A field that passes its format check and resolves against a register does not need review at 0.6 confidence; a free-text field with no constraint at 0.9 might. Applying a single confidence threshold across all fields ignores that some of your fields are verifiable and some are not, and it sends the wrong work to people.

Where a constraint fails, the confusion-set repair is worth trying before escalating: handwriting substitutions are systematic — a 1 read as a 7 or as a slashed 7, a 4 as a 9, a 5 as an S, a 0 as a 6 — so generating the small set of variants and testing each against the constraint often produces exactly one candidate that passes. One candidate is a repair; two is an ambiguity a person should see.

The failures that are specific to handwriting on forms

These are the ones that survive a good recogniser and a good region model, and each needs its own handling:

• Overflow. A long name written into a short box continues past its right edge and over the next field. Region-based cropping cuts it off, so the value is truncated with high confidence. Detect ink crossing a region boundary and widen, or flag.
• The blank that is answered anyway. A model given a crop of an empty box will sometimes produce a value. Ask for an explicit empty result, and cross-check with a cheap ink-density measurement on the region — near-zero ink and a non-empty value is a contradiction worth trusting the pixels on.
• Corrections. A value struck through with the replacement written above, beside, or in the margin. Both values are in the region and the wrong one is often the more legible. Treat a region containing a strike as a field you decline to read rather than picking.
• Marks that are not ticks. People circle the label instead of ticking the box, cross the box out to mean yes, or tick between two boxes. A checkbox detector looking only inside the box reports the field as blank. Evaluate the label region as well as the box, and report an ambiguous mark as ambiguous.
• Ditto and “same as above”. A mailing address given as a stroke or a phrase referring to another field. It is a reference, and resolving it is a documented rule you apply after extraction, not something to let a model quietly interpolate.