Extracting Damage Descriptions From a Vehicle Inspection Report

A vehicle condition report records damage twice: as marks on a line drawing of a car, and as written notes. The marks carry information the notes do not — chiefly where the damage is — and that information is encoded entirely in pixel position. Extraction here is a geometry problem with a text join on top.

Two records of the same damage

The forms vary by fleet operator, auction house, rental company and leasing firm, but the pattern is stable. Somewhere on the page is an outline of a vehicle — usually four views: driver side, passenger side, front, rear, sometimes a roof plan. The inspector marks damage on the outline with a symbol or a code, and separately writes notes in a table or a free-text block.

The two records are not redundant and neither is complete. The diagram gives location precisely and description barely. The notes give description and severity and often no location beyond a panel name. A claim, a lease-end charge or a resale listing needs both, so the extraction target is a single damage[] array in which each item records which sources contributed to it:

{
  "marker_id": "4",
  "panel": "driver_front_door",
  "view": "driver_side",
  "position": { "x": 0.41, "y": 0.62 },   // normalised to the diagram
  "type": "dent",
  "severity": "moderate",
  "dimension": { "value": 3, "unit": "inch" },
  "sources": ["diagram", "narrative"],
  "narrative_text": "4. Dent driver front door, approx 3in, below trim line"
}

sources is the field that makes the output honest. A damage item present only on the diagram has no description; one present only in the notes has no verified location. Both are legitimate outputs and a consumer needs to know which it has.

A mark’s meaning is its position

The mark itself is nearly contentless — a circle, an X, a two-letter code. Its meaning is “there is damage of this type here”, and “here” is a coordinate. So the pipeline has to recover geometry, and asking a model to describe the image will not do it: a description like “there are marks on the driver side of the vehicle” has discarded the only thing the diagram was for.

The steps that work:

• Locate the diagram region on the page and crop it. Diagrams are printed at a known place on a known form, so on a fixed-template workflow this is a constant rather than a detection problem; on a mixed-template one, detect the outline by its distinctive line art.
• Normalise coordinates into the diagram’s own space, not the page’s. Scans are rotated, skewed and scaled differently; a mark at page coordinate (412, 883) means nothing across two documents, while a mark at 0.41 of the diagram’s width and 0.62 of its height means the same thing on both.
• Map the normalised point onto a panel using a template — a set of named polygons over the reference outline. This is where “x=0.41, y=0.62” becomes “driver front door”, and it is a lookup rather than an inference, which means it is auditable and does not vary run to run.

Do the deskew before the normalisation. A three-degree rotation across a diagram the width of the page moves a mark by a substantial fraction of a panel, which is the difference between a door and a quarter panel, which is the difference between two repair estimates.

Joining the diagram to the narrative

On forms that number their marks, the join key is the number: a circled 4 on the diagram corresponds to line 4 in the notes table. This is the cleanest case and it has a specific weakness — the join key is a single handwritten digit inside a circle, which is among the hardest things on the page to read. A misread 4 as 9 does not produce an error; it produces a dent silently relocated to the other side of the car.

Two defences. First, the join is a bijection: every diagram marker should have exactly one narrative line and vice versa, so check the two sets against each other and treat any unmatched marker or orphan lineas a review item. A duplicate number is a certain misread. Second, where the narrative line names a panel, compare it against the panel derived from the marker’s coordinates. Agreement confirms the join; disagreement identifies which record to look at. Neither check costs a model call.

On forms that do not number their marks, there is no join key and you should not manufacture one. Proximity matching — pairing a note about a door with the nearest marker to a door — is a guess dressed as data. Output the two sets separately, flag the document as unjoined, and let the consumer decide. Coded marks (a two-letter damage code written at the mark rather than a number) sit in between: the code gives you a type to match on, so a note describing a scratch can be joined to a scratch code with reasonable confidence, but only where the codes are unambiguous within the view.

Do not invent a code table. Damage code sets are defined by the operator or the auction house that printed the form, and the legend is normally on the form itself. Extract the legend and use it; where the form has no legend, record the raw code rather than mapping it to a vocabulary you assumed.

The VIN check digit

The header of the report carries a vehicle identification number, and for vehicles built to the North American requirement it carries a check digit you can verify. The rule, set out in the United States in 49 CFR Part 565 and published through the eCFR, is: transliterate each of the seventeen characters to a value, multiply by a positional weight, sum, take the remainder modulo 11, and that remainder is the ninth character — written as X when it is 10.

weights, positions 1..17:
  8 7 6 5 4 3 2 10 0 9 8 7 6 5 4 3 2
                    ^ position 9 is the check digit itself

digits transliterate to themselves
letters: A=1 B=2 C=3 D=4 E=5 F=6 G=7 H=8
         J=1 K=2 L=3 M=4 N=5 P=7 R=9
         S=2 T=3 U=4 V=5 W=6 X=7 Y=8 Z=9

I, O and Q are not used in a VIN at all

Two things fall out of this for extraction. The check digit gives you the same arithmetic rejection an ISBN does — a misread VIN almost never checks out. And the absence of I, O and Q means any of those characters in an OCRed VIN is definitionally an error: map I to 1, O to 0 and Q to 0, re-run the check, and a large share of failures resolve without a human. Where a vehicle was built outside the North American scheme the check digit may not be present, so treat a failure as a flag rather than a rejection, and record which regime you assumed.

Where it breaks

The photographs are the real report. Modern condition reports are mostly images with a form wrapped around them, and the location of a damage photo in the sequence is often its only label. Extract the photo captions and their ordering; a photo with no caption adjacent to a numbered note is probably that note’s evidence, and probably is the right word to store.

Pre-existing versus new damage. Rental and lease inspections happen at check-out and check-in, and the same form is used for both. A damage item is only meaningful next to which inspection recorded it. Extract the inspection type and timestamp as document-level fields and refuse to emit damage records without them.

Odometer units and digits. A reading of 12345 may be miles or kilometres, and the unit is usually a printed label rather than part of the value. Trip meters and total meters both appear. Extract the unit explicitly and treat an unlabelled reading as unit-unknown.

Checkbox grids for mechanical items. Below the diagram, most forms have a grid of components with pass, fail or not-applicable boxes. These are a different extraction problem — small marks in a dense grid where the alignment between a tick and its row determines the answer, and where an ambiguous mark is common. The severity and triage reasoning that applies to the results is closer to a home inspection report than to the damage diagram above it.