Extracting Container and Cargo Details From a Shipping Manifest

A manifest is a long table that adds up. That single property is worth more than any confidence score, because it turns a silent extraction failure into a loud one — but only if you can tell a data row from a subtotal row, and the two look identical.

The shape of a manifest

A cargo manifest is a header block naming the vessel, voyage, ports and filing party, followed by a repeating line-item block. Each line typically carries a bill of lading number, container numbers and seal numbers, marks and numbers, a package count with a package type, a goods description, a gross weight and a measurement in cubic metres. The header block is small and prose-like; the line block is large and rigidly columnar, and they fail in completely different ways.

The bill of lading number is the closest thing to a key. In United States filings it is conventionally a four-letter Standard Carrier Alpha Code followed by the carrier’s own reference, so the SCAC prefix is a cheap format check and a cheap way to spot a line that was merged with its neighbour. It is not a checksummed identifier the way a container number is, so treat a well-formed one as plausible rather than verified.

The footing check

Manifests state totals: total packages, total gross weight, total measurement, sometimes total containers. The check is a cross-field amount validation rule: the sum of the extracted lines equals the stated total, within a tolerance that accounts for the document’s own rounding.

The tolerance matters more than it looks. If line weights are printed to three decimal places and the total is printed to none, the sum of forty lines can differ from the printed total by several units purely from rounding, and a strict equality test will fail every long manifest. Compare with a tolerance derived from the printed precision: half a unit of the last printed decimal place, times the number of lines, is a defensible bound. Anything outside that is a real discrepancy.

When the check fails, the direction of the failure tells you what went wrong. A sum that is short by roughly the size of one line means a line was dropped, usually the first or last on a page. A sum that is short by a large round amount often means a whole page was missed — worth checking against the printed “page n of m”. A sum that exceeds the total almost always means something was counted twice, and the thing counted twice is almost always a subtotal.

Brought forward, carried forward

This is the specific trap, and it is invisible to a model asked to “extract every row of the table”. Multi-page manifests carry running subtotals across the page break: the last row of a page is a carried-forward figure, and the first row of the next page repeats it as brought forward. They occupy the line-item columns. They have a weight in the weight column. They look exactly like cargo.

Sum every numeric row on a three-page manifest whose true total is 28,335 kilograms and you can get this:

page 1  lines                    12,480.000
        C/F                      12,480.000
page 2  B/F                      12,480.000
        lines                     9,215.000
        C/F                      21,695.000
page 3  B/F                      21,695.000
        lines                     6,640.000
        TOTAL (printed)          28,335.000

naive sum of every numeric row:  84,205.000
true sum of cargo lines:         28,335.000

The good news is that the error is enormous rather than subtle, which is exactly why the footing check is worth running: a three-times overshoot is unmistakable, whereas a single dropped line is not. The fix is to classify rows before summing them. Subtotal rows have identifiable signatures — an empty bill of lading number, an empty package count, a description column containing only a marker such as B/F, C/F, BROUGHT FORWARD or SUB TOTAL, and a position at the very top or very bottom of a page’s table region.

The most robust of those signals is the empty key. A row with a weight but no bill of lading number is not a shipment. Make the key field required in the row schema and give the model a separate, explicitly named place to put subtotal rows, so that discarding them is a decision recorded in the output rather than an omission.

Package counts are not one number either

“1240 CTNS ON 62 PLTS” is one cell containing two counts and two package types, and which of them is “the” package count depends on who is asking. The outer packaging is what the carrier counts and what the manifest total usually foots to; the inner count is what the commercial documents describe. Extract quantity and package type as a repeating pair rather than as a number and a string, and expect more than one pair per line.

Package type codes are standardised — the UN/CEFACT recommendation 21 codes such as CT for carton, PX for pallet and BG for bag — but manifests print free text at least as often as codes, and the free text is abbreviated by whoever typed it. Normalise to the code list where you can match confidently and keep the original string where you cannot, because an unmatched abbreviation is information and a wrongly matched one is not.

What breaks across a page break

• Repeated headers. Every page reprints the column headings. Extracted as a row, a header contributes a line with no numbers and a description of “GROSS WEIGHT”.
• Split rows. A line whose description or marks run to four physical lines can straddle the page boundary, leaving half a shipment at the bottom of one page and half at the top of the next. Detect it by the missing key on the continuation half and join, do not treat the fragments as two shipments.
• Shifted columns. Column x-positions are not guaranteed to be identical on every page of the same document, especially when the manifest was generated by concatenating filings. Derive column boundaries per page rather than once for the file — a layout problem covered properly in the page on column misalignment across pages.
• Embedded newlines in marks and numbers. The marks column routinely contains several lines of stencilled text with no fixed format. Collapsed into one string it is harmless; allowed to drive row segmentation, it splits one shipment into five.
• Continuation sheets with no total. Not every manifest prints a grand total. When there is none, the footing check is unavailable and the next best structural check is contiguity of line numbers, if the document numbers its lines at all.