Extracting Line Items From a Handwritten Purchase Order

Handwritten text and handwritten numbers are different problems. A model reading a handwritten word has a vocabulary to fall back on. Reading a handwritten quantity, it has nothing — every digit string is a legal digit string, and a wrong one is indistinguishable from a right one.

Digits have no spell-check

Recognition of handwritten prose works partly because language is redundant. Misread two letters in a word and context restores it. That entire mechanism is absent for a quantity field. If a written 7 is read as 1, the result is a perfectly plausible order for one unit instead of seven, and no amount of context in the surrounding text says otherwise.

The confusable pairs are well known to anyone who has processed forms: 1 and 7 (especially where a continental crossed seven is written without its bar, or where a 1 is written with a long serif), 4 and 9 when the 4 is closed at the top, 0 and 6, 3 and 8 when written quickly, 5 and S, 2 and Z, and 6 and b. On a fax or a low-contrast scan, a closed 4 and a 9 can be genuinely identical.

The practical consequence is that per-character confidence, if your pipeline exposes it, is far more useful on a handwritten PO than on a typed one, and field-level confidence is far less useful. A quantity of “17” where the 7 is uncertain is a completely different risk from one where the 1 is uncertain, and a single number for the field hides which.

The invisible decimal point

The most expensive handwriting failure is not a digit at all. It is a decimal point that did not survive scanning. A unit price written 12.50 reads as 1250, and the error is a factor of one hundred — large enough to matter enormously and, crucially, still a plausible price for something.

Three related cases. A comma used as the decimal separator is thinner than a point and disappears more readily. A price written with a trailing dash for the minor units, as in 12,—, means twelve and zero cents, which a parser will not know. And thousands separators are frequently omitted by hand, so 1250 may be twelve fifty or twelve hundred and fifty depending only on a mark that may not be there.

Never let the parser resolve this by preference. Resolve it with the arithmetic on the form, which is what the next two sections are about, and where the arithmetic cannot resolve it, send it to a human. A hundred-fold price error that reaches a purchase order is not a data quality statistic; it is a purchase.

Why fluency hurts here

A vision-language model does not read a form character by character. It produces a plausible continuation given everything it has seen, and for handwriting this is usually an advantage — it is what lets it read a scrawled product description correctly. On numbers, the same mechanism is a liability, and in a specific direction: the model tends to produce round, ordinary, plausible values. An ambiguous quantity resolves to 10 rather than 13; an ambiguous price resolves toward a common price point; a partially obscured total resolves to something that looks like a total.

This is the same class of behaviour described in vision hallucination and it is worse on forms than on photographs, because a form is highly structured and the model has strong expectations about what belongs in each cell. Two mitigations follow directly.

• Ask for the mark, not the meaning. Prompt for the characters as written, with an explicit instruction to return a null and a note where a character is illegible rather than to infer it — handling an illegible field as its own outcome rather than as a low score. “Do not complete or correct a number” is a load-bearing sentence in a handwriting prompt.
• Do not give the model the totals when reading the lines. If the model can see the order total while transcribing quantities, it will reconcile them for you — silently, by adjusting a quantity. You want the two readings independent, precisely so you can compare them.

The check the form gives you free

A purchase order form almost always has more numbers than it needs. Quantity, unit price and line total are all written, and the third is the product of the first two. That redundancy is a constraint solver for the ambiguity above.

Suppose a line reads: quantity ambiguous between 12 and 17, unit price clearly 14.50, line total legibly 174.00. Then 12 times 14.50 is 174.00 and 17 times 14.50 is 246.50. The quantity is 12, decided arithmetically, with no confidence score involved. Now suppose the price is ambiguous between 12.50 and 1250, quantity 8, line total 100.00: only the first multiplies out. The invisible decimal point is recovered.

// resolve one uncertain field from the other two
function resolve(candidates: number[], other: number, total: number,
                 toleranceMinor = 1): number[] {
  return candidates.filter(
    (c) => Math.abs(Math.round(c * other) - total) <= toleranceMinor
  );
}

// qty ∈ {12, 17}, unit 1450 minor, line total 174000 minor
resolve([12, 17], 1450, 174000);   // -> [12]

The order of operations matters: transcribe all three fields with their alternatives first, then solve, rather than committing to a reading field by field. Where the solver returns more than one survivor, or none, you have learned something concrete — none means at least two of the three readings are wrong, which is a much stronger signal than any of them being individually uncertain. The column total gives a second, weaker constraint over the whole page.

What to do with what is left

Some fields will not resolve, and the design question is what a partially confident PO becomes. A few rules that hold up:

• Route by field, not by document. A page where only one quantity is uncertain should reach a reviewer as one cell to confirm, not as a form to re-key. That is the difference between a review queue that survives and one that gets abandoned; see showing confidence to a reviewer.
• Never auto-approve a numeric field on model confidence alone. The arithmetic check is the gate. Confidence chooses what to check first, and only once it has been calibrated against outcomes.
• Treat crossings-out as data. An overwritten quantity has an old value and a new one, and a model asked for “the quantity” may return either. Ask explicitly for struck-through or amended values as a separate field so a human can adjudicate.
• Units are part of the quantity. “2 doz”, “3 ctn”, “5 pr” are common on handwritten forms. Extract the unit of measure as its own field; multiplying it out is a business decision that depends on the item master.

The general techniques for reading handwriting are covered in handwriting recognition with language models. What is specific to a purchase order is that the document is arithmetic all the way down, and arithmetic is a stronger check than recognition will ever be.