Extracting Structured Data From a Handwritten Inventory Count Sheet

A vision model’s confidence score on a handwritten “7” tells you very little. What tells you a great deal is that the sheet says 18, the system says 78, and one digit substitution explains the difference exactly. On a count sheet the check is arithmetic against another system, not a probability the model reported about itself.

What makes a count sheet hard

Physical inventory counts are recorded on paper because the warehouse is cold, the scanner battery is flat, or the process predates anyone currently employed. The resulting artefact is a pre-printed grid with handwriting in it, and it has several properties that no amount of handwriting recognition quality fixes.

The most important is that the sheet is usually a cycle count form pre-printed with the SKU, the description, the location and — on badly designed forms — the expected quantity from the system. So the page contains two kinds of marks: printed text that is already in your database, and handwritten marks that are the only new information on the page. An extraction that reads the quantity column without distinguishing printed from handwritten strokes can return the system’s own expected figure and report perfect agreement, which is the worst possible failure because it is invisible.

If your forms carry the expected quantity, the fix is at the form level rather than the model level: blind counts do not print it. Where you cannot change the form, extract the printed and handwritten values as two separate fields and treat their equality as a fact to be recorded rather than as a confirmation.

The other structural properties: quantities are often written in units that are not eaches, corrections are made by striking through and rewriting rather than erasing, counters use tally marks for small counts, and a second recount column may be filled in by a different person in different handwriting.

Blank is not zero

This is the single most consequential rule on the page. A blank quantity cell means the SKU was not counted. A written “0” means it was counted and there is none. If extraction maps both to 0, the inventory adjustment will write off every uncounted line, and the error is silent because the resulting record is well-formed.

So the quantity field is nullable and the record carries a status: counted, not_counted, illegible, or ambiguous. Four states, of which illegible is the one most often collapsed into the others, and a schema with a non-nullable integer quantity cannot represent three of them. Constrain the model to that enumeration explicitly rather than hoping; this is exactly the kind of contract that strict structured output exists to enforce.

Struck-through corrections are a related case with a positional answer. Where a cell contains two numbers and one has a line through it, the surviving value is the one not struck; where both are written and neither is struck, the lower or later-written one is not reliably the intended value and the cell is ambiguous. Do not let the model pick. A count sheet cell with two numbers in it is a question for the counter, and the counter’s initials are on the sheet.

The variance check does the work

Every counted line has a companion figure in the warehouse system: the perpetual on-hand quantity for that SKU at that location. Comparing them turns extraction quality into a measurable, classifiable quantity.

variance      = counted - system_on_hand
variance_pct  = variance / system_on_hand      (system_on_hand > 0)

SKU       system   counted   variance   variance_pct
--------  ------   -------   --------   ------------
A-10422       78        18       -60        -76.9%
B-20817       72         6       -66        -91.7%
C-30155      144       143        -1         -0.7%
D-41003        0        12       +12             n/a

Those four rows are four different findings and only one of them is an inventory problem. Line C is an ordinary small discrepancy: one unit, entirely plausible as a genuine miscount or shrink, and it goes to the normal adjustment process. Line D is a count against a SKU the system thinks is empty, which is usually a location error rather than a quantity error.

Lines A and B are extraction findings dressed as inventory findings. Line A is 18 against 78 — a single leading-digit substitution of 1 for 7, which is the most common handwriting confusion there is, particularly where a European crossed seven is written by one counter and read by a system trained mostly on unbarred ones. Line B is 6 against 72, and 72 divided by 6 is exactly 12: this SKU is packed 12 to a case, and the counter recorded cases where the sheet expected eaches.

Both are checkable without a human. If system_on_hand divided by counted equals the SKU’s case pack exactly, emit suspected_uom_mismatch rather than a variance. If a single-digit edit to the counted value lands within a small percentage of the system figure, emit suspected_misread with the candidate value attached. Both go back for confirmation, but they go back with a hypothesis, and confirming a hypothesis takes a fraction of the time recounting a bay does.

Digit confusions have structure

Handwritten digit errors are not uniformly distributed, and knowing the shape of them is what makes the candidate-generation step above practical rather than a brute-force search:

• 1 and 7 — the crossbar convention varies by country, and a European 1 with a long upstroke reads as a 7 in isolation.
• 4 and 9 — a closed-top 4 is a 9 with a short tail.
• 3 and 8, 5 and 6, 0 and 6 — all differences of one closed loop, which a hurried pen closes or leaves open.
• 2 and 7 — in fast writing, particularly with a flat-bottomed 2.
• Trailing zeros — a 10 read as 100 or the reverse. This one is worth special handling because it is an order of magnitude, and any variance that is a factor of exactly 10 or 100 should be flagged as a suspected digit-count error before it is treated as shrink.

Generate candidates by applying these substitutions to the read value, plus insertion and deletion of a trailing zero, and test each against the system figure. This is a handful of comparisons per line and it converts an unexplained variance into a named, reviewable hypothesis.

Two more sheet-level checks are cheap and catch different things. If the sheet has a printed total line, foot the extracted quantities against it. And if the same SKU appears on two lines — which happens when stock is in two locations, or when a counter wrote a line twice — decide by location, and flag a duplicate SKU with an identical location as a probable double entry rather than summing it.

Putting it together

1. Extract per line: printed SKU, printed location, printed description, handwritten quantity, handwritten unit of measure if present, recount value if present, and the counter’s initials. Keep printed and handwritten fields separate throughout.
2. Assign each line a status of counted, not_counted, illegible or ambiguous. Never coerce a blank to zero.
3. Join to the system of record on SKU and location, bringing back on-hand quantity, case pack and unit of measure.
4. Compute the variance, then classify it: suspected_uom_mismatch when the ratio equals the case pack, suspected_misread when a single-digit or trailing-zero edit reconciles it, location_mismatch when the system holds zero, and genuine_variance otherwise.
5. Route by classification, not by model confidence: hypotheses go to the counter for confirmation, genuine variances above a value threshold go to a recount, and small genuine variances go straight to adjustment. The sampling rate for review then applies only to the lines that passed every check, as a check on the checks.