Extracting Checked and Unchecked Boxes From a Scanned Form

Detecting a ticked box is easy: there is a mark and you can see it. Detecting an unticked box means asserting that nothing is there, and nothing looks identical to a faint pencil stroke, a box that fell off the edge of the scan, and a box that was never on the form. That asymmetry is the entire problem.

The unchecked box is the hard one

A checked box requires one detection. An unchecked box requires two: the box is present at a known location, and there is no mark inside it. Skip the first and “unchecked” silently absorbs every failure mode in the pipeline — a page that scanned short, a region occluded by a staple, a box your parser never looked at. Each of those produces a confident “no” on a form where the applicant said yes.

This is also why asking a vision model “which boxes are checked?” underperforms so badly on exactly the cases that matter. The model has no representation of a box it failed to look at, so it cannot report one. It is drawn toward the labels rather than the geometry, and it will happily report the option that is semantically plausible given everything else on the form. And it is unreliable at the one answer you most need to be able to trust, which is “none of them”.

The fix is to make the box inventory come from somewhere other than the scan. If the form is a known template — and on a fixed-layout form it almost always is — register the scan against a blank original, and you then know how many boxes exist and where. “Box not found” becomes a loud error rather than a silent unchecked, and the count check is free: found boxes must equal template boxes.

Find the box, then measure the ink

1. Deskew and de-warp. A page scanned at two degrees of rotation puts a rectangular crop over the corner of the neighbouring box. Correct rotation before any coordinates are used, and detect page curl on photographed forms, where the distortion is not affine.
2. Normalise illumination. Estimate the background with a large-kernel blur or morphological closing and subtract it. This removes the gradient from a fold, a shadow, or an uneven platen before thresholding rather than after, which is the difference between a policy that works near a binding and one that does not.
3. Locate box geometry. Either from the registered template, or from the image: small near-square closed contours in a narrow size band, whose sides are axis-aligned after deskew. Reject candidates outside the expected size range — a table cell is not a checkbox.
4. Threshold locally, not globally. A single global threshold tuned to printed text erases pencil. Use a local adaptive method so the decision at each pixel is relative to its own neighbourhood.
5. Measure the interior. Erode the box inward by a couple of stroke widths so the printed rule itself is excluded, then compute the fraction of dark pixels in what remains. Apply a morphological opening first to remove isolated speckles smaller than a pen stroke, which is what dust and scanner noise look like.
6. Decide with two thresholds. Below the low one, unchecked. Above the high one, checked. Between them, review. One threshold forces every marginal case into a confident answer, which is precisely the population you wanted to catch.

The ink fraction is a measurable quantity with a physical meaning, which is why this beats a model score for this particular decision: you can look at the distribution across a batch and see the two modes and the valley between them, and set the thresholds where the data says rather than where a default says.

The confusions, and a policy for each

Light pencil marks

A pencil tick can carry a fraction of the ink density of a ballpoint one, and a global threshold set for printed text removes it entirely, producing a confident unchecked. Local adaptive thresholding recovers most of it. Beyond that, compute the form’s overall foreground density and, on a page that is uniformly faint, shift both thresholds down together rather than lowering only the high one — a faint page is faint everywhere, so the ratio is more stable than the absolute level.

An X instead of a tick, or a fill, or a circle

For the ink measurement none of these matters: all are marks and all raise the fraction. The genuine ambiguity is semantic. On most forms a cross means selected. On some — particularly forms that instruct the respondent to strike out what does not apply — a cross means the opposite, and a circle drawn around a box rather than inside it is a mark that the interior measurement will miss entirely. Pixels cannot settle this. Read the form’s own instruction line, record which convention this template uses as a property of the template, and measure a small annulus outside the box as well as the interior so a circling respondent is detected rather than silently unchecked.

Boxes near a fold or the binding

A fold casts a dark band that raises the ink fraction and produces false positives; a binding shadow does the same and also warps the geometry so the box is no longer square. Background subtraction removes most of the band. For the rest, detect the fold explicitly — it is a long, low-frequency, roughly straight dark structure spanning the page — and force any box within a small distance of it into the review band regardless of its measured value. A rule that says “this measurement is not trustworthy here” is more honest than a threshold adjustment that pretends it is.

Bleed-through from the reverse side

Duplex scanning of thin paper shows the reverse faintly, and the reverse of a form is usually another form with boxes in structurally similar positions. The bleed is fainter and horizontally mirrored relative to the front, so the low threshold catches most of it. Where it does not, comparing against the reverse image at the mirrored coordinate is decisive: a mark that has a much stronger counterpart on the other side at the mirror position is bleed.

Both boxes in an exclusive pair marked, or neither

Neither of these is a detection failure and neither should be resolved by picking. Both marked usually means the respondent changed their mind without striking out the first; neither marked in a mandatory field means the form is incomplete. They are findings about the document, and the schema has to be able to represent them or they cannot be reported.

What the scanner did to the image

Before any of the above, the scan itself may have altered the content. Some office scanners compress bitonal documents with a pattern-matching scheme — JBIG2 in its lossy mode — that stores one representative image for each cluster of visually similar marks and reuses it. In 2013 David Kriesel documented and publicised cases where this substituted digits in scanned documents on widely deployed multifunction devices, so the scan showed numbers the paper did not.

A checkbox is a small, highly repetitive glyph, which is exactly the kind of pattern such a scheme clusters aggressively. Do not accept a lossily compressed scan as evidence for anything consequential. Require lossless bitonal compression or a greyscale original, check what the device is configured to produce rather than assuming, and record the compression found in each file alongside the extraction — it is a property of the evidence, not of the pipeline. Kriesel published his findings and correspondence.

Three states and one contradiction

{
  "template_id": "claim-form-v4",
  "page": 2,
  "boxes": [
    { "field_id": "q7_option_a", "state": "checked",
      "ink_fraction": 0.41, "interior_px": 1840,
      "bbox": [412, 1290, 34, 34], "source": "template_registration",
      "flags": [] },
    { "field_id": "q7_option_b", "state": "unchecked",
      "ink_fraction": 0.02, "interior_px": 1836,
      "bbox": [412, 1338, 34, 34], "source": "template_registration",
      "flags": [] },
    { "field_id": "q7_option_c", "state": "review",
      "ink_fraction": 0.11, "interior_px": 1841,
      "bbox": [412, 1386, 34, 34], "source": "template_registration",
      "flags": ["near_fold"] },
    { "field_id": "q8_option_a", "state": "not_observed",
      "ink_fraction": null, "interior_px": null,
      "bbox": null, "source": "template_registration",
      "flags": ["box_not_located"] }
  ],
  "group_findings": [
    { "group_id": "q7", "finding": "multiple_marked_in_exclusive_group" }
  ]
}

not_observed is the value the whole page exists for. It is not a null and it is not unchecked; it says the pipeline could not make the second detection, so no claim about the box is being made. Keeping ink_fraction and interior_px alongside the state makes every decision re-derivable: change a threshold and you can recompute the entire archive without re-reading a single image, which you will want to do the first time a batch arrives from a different scanner.

Everything above concerns one box. Turning a page of boxes into answers — which options belong to which question, what an empty group means, how to represent a check-all-that-apply alongside a pick-one — is a schema design problem, and it is where the three states have to survive contact with your data model.