Extracting Structured Answers From a Filled-In Exam Answer Sheet

An answer sheet is two documents printed on one page. The bubble grid should never reach a language model at all; the free-response section has to. Treating them as one extraction is why the naive pipeline produces scores that are wrong in ways nobody notices.

One document, two entirely different problems

The multiple-choice section is a fixed template. The positions of every response bubble are known before the sheet is printed, because the sheet was printed from that template. The only unknown is how dark each of those known positions is. That is a measurement problem with a deterministic answer, and it has been solved by optical mark recognition for decades without any machine learning at all.

The free-response section is the opposite: unknown content, handwritten, in an unknown amount, and genuinely a reading problem. It is where a vision model earns its cost.

The mistake is to photograph the whole sheet and ask a model what the student answered. A model asked to read a grid of two hundred bubbles has to recover the row and column correspondence from the image itself, and it will drift — most often part-way down a long column, where nothing in the local image distinguishes item 43 from item 44. The failure is not a refusal or a low-confidence output. It is a confident, complete, systematically shifted set of answers.

The grid is a coordinate system, not a picture

Printed answer sheets carry registration marks — corner targets, or a column of timing marks down one edge — for exactly this reason. They define a coordinate frame in the scanned image, and the template maps every item and option to a cell in that frame. The processing chain is mechanical.

1. Locate the registration marks and compute the transform from template coordinates to image coordinates. A page scanned a degree or two off square moves cells by more than a bubble radius near the far edge, so deskewing before sampling is not optional.
2. For each item and option, sample the cell and compute a fill ratio — the proportion of dark pixels inside the bubble outline, with the printed outline itself excluded so an empty bubble scores near zero.
3. Classify each cell against thresholds, then classify the item from its option cells.
4. Emit one record per item number from the template, including items with no mark, so the output has a fixed length known in advance.

Two production details cause more trouble than the algorithm. Forms are often printed in a dropout colour intended to vanish under a scanner configured to drop it; scan the same form on a general-purpose scanner that keeps the colour and every cell contains printed ink, so every fill ratio rises and a fixed threshold marks everything. And a sheet photographed with a phone rather than scanned has perspective distortion as well as rotation, which a four-point registration transform handles and a rotation-only correction does not.

The offset failure that produces a plausible score

This is the failure worth designing the whole pipeline against. If answers are collected as a list in the order they were read, a single unanswered item that produces no entry shifts every subsequent answer up by one position. Item 18’s response becomes item 17’s, and so on to the end of the section.

Nothing about the result looks wrong. The list is well-formed, the values are all valid options, the count is only one short, and a candidate who left one question blank is entirely ordinary. The score is simply wrong, for every item after the blank, and it is wrong in a way that will correlate across a whole batch if the scanning was consistent.

The defence is structural rather than statistical: the item number must come from the template cell, never from the position in a list. An unanswered item emits a record with a null response, so the output length is the item count regardless of what the student did. Then assert it — a section of sixty items that produces fifty-nine records is rejected before scoring rather than after. This is a schema-level invariant of the kind schema edge cases discusses generally; here it is the difference between a correct score and an actionable, plausible, wrong one.

Four states, including two marks

A cell is not marked or unmarked. Using illustrative thresholds — the right values depend on the form, the printing and the scanner, and are set by calibrating against sheets whose answers are known — a design that works treats a fill ratio above roughly 0.6 as marked, below roughly 0.2 as blank, and anything between as ambiguous. An incomplete erasure typically lands in that middle band, which is precisely why the band must exist rather than being split by a single threshold.

At the item level that yields four outcomes, and each needs a distinct value in the output.

• Exactly one marked cell. The ordinary case.
• No marked cell. An omission, which is a real answer state and not missing data.
• Two or more marked cells. Flag it. Do not resolve it by taking the darkest, and do not resolve it at all — whether a multiple mark scores as incorrect, as omitted, or is referred to a proctor is a rule belonging to the examination, not to the extractor. Resolving it inside the pipeline hides a decision that has to be defensible when a candidate appeals.
• One marked and one ambiguous, or all ambiguous. Human review, with the cropped cell image attached so the reviewer decides in seconds — the general pattern is review queue source highlighting, and reviewing an ambiguous mark is one of the few review tasks where a crop is genuinely sufficient evidence.

Record the fill ratios themselves, not just the decision. When a batch turns out to have been scanned with the wrong settings, the ratios let you reclassify without rescanning — the same argument as correcting a single field without a rerun. The decisions do not.

Free response, page identity and the student ID grid

Free-response pages bring back the ordinary document problems, plus one that is specific to examinations: attaching the right pages to the right candidate and the right question.

• Page identity is printed, so use it. Answer booklets carry a page identifier, often as a barcode or a small printed code including the form version and page number. Scanned batches arrive out of order, duplex pages arrive interleaved, and a jam causes a re-feed. Sorting by scan order is guesswork; sorting by the printed identifier is not.
• The student identifier is itself a bubble grid. A misread there attaches an entire paper to the wrong candidate, which is a far worse error than any single item. Where the sheet carries both a bubbled identifier and a printed or barcoded one, check them against each other and against the roster, and treat a disagreement as blocking.
• Blank is a meaningful answer. A page containing no writing must be recorded as answered-blank, not skipped, or the offset problem returns in a different form.
• Work in the margin. Candidates continue answers in margins, on the reverse, and on additional sheets. A crop tightly fitted to the answer box loses it, and a marker who later sees the paper will notice.
• Do not score in the extraction step. The extractor reports what is on the page. Comparing to a key, applying negative marking, and handling a multiple mark are scoring policy, and keeping them separate is what lets a rescore run without a rescan.