Document Understanding: PDFs, Tables and Layout

Models handle documents well enough that the remaining failures are concentrated and specific. Almost all of them are the same failure: a two-dimensional structure has to become a one-dimensional token sequence, and some structures do not survive the flattening. Here is the list, with what the wrong output looks like.

Two ways a PDF reaches the model

Before debugging anything, know which path you are on, because they fail differently.

• Text extraction. A library (pdfplumber, PyMuPDF, pdftotext) pulls the embedded text layer. It is exact when there is one, free, and throws away every visual cue — a bold header and body text arrive identical. It also emits nothing for a scanned page, which is the classic silent failure: you send an empty string and the model answers from thin air.
• Page as image. Render each page and send the pixels. Layout, rules, shading and handwriting all survive; small type may not, and you pay image tokens per page.

A cheap guard for the first path, worth writing before anything else: if the extracted text for a page is under a few dozen characters, treat the page as scanned and switch paths. That one branch catches a large share of production incidents in document pipelines.

Some APIs now accept a PDF directly, which sounds like a third path and is really a convenience wrapper over one of the first two — the service renders the pages and tokenises them, or extracts the text layer, and which one it did determines what you are billed and what the model sees. It is worth reading that behaviour in the provider’s documentation rather than assuming, because the difference between “40 page images” and “12,000 tokens of extracted text” is a factor of five on the invoice and a complete change in what layout information survived. Where you care, render the pages yourself: you then control the resolution, and you can crop.

Failure: merged and spanning cells

A header that spans two columns has no representation in markdown, so it gets duplicated, dropped, or promoted into a fake row. Consider a perfectly ordinary financial table:

                |      2025      |      2024      |
  Region        |  Q1  |   Q2   |  Q1  |   Q2   |
  ---------------------------------------------------
  EMEA          | 1.2  |  1.4   | 1.1  |  1.0   |
  Americas      | 2.0  |  2.2   | 1.9  |  2.1   |

flattened to markdown, a common wrong result:

| Region   | Q1  | Q2  | Q1  | Q2  |
|----------|-----|-----|-----|-----|
| EMEA     | 1.2 | 1.4 | 1.1 | 1.0 |

The year is gone. Two columns are now called Q1 and any
consumer that keys by header name silently takes the first.

The fix is to stop asking for markdown. Ask for a JSON schema that can hold the structure — a list of records with explicit year and quarter fields — so there is nowhere for the spanning header to go except into the data. If you must have a grid, ask for HTML rather than markdown: colspan exists there, and the research datasets for this task (PubTables-1M, FinTabNet) encode ground truth as HTML for exactly that reason.

Failure: tables that cross a page

Page 4 ends mid-table; page 5 repeats the header row and continues. If you process pages independently you get two tables, the second of which may lose its first data row to a header misidentification. If you concatenate blindly you get a header row sitting in the middle of your data, which then becomes a record whose numeric fields contain the strings “Q1” and “Q2”.

Neither is fixed by a better model, because the model was never shown both pages together. Overlap the window — send page n and n+1 in the same request when a table touches the bottom margin — and post-process by dropping any extracted row whose values equal the header row.

Failure: numbers that are not numbers

Financial and scientific documents are full of notation that means something, and flattening strips the meaning while keeping the digits.

Every one of these is fixed in the schema rather than in the prompt. Ask for the raw string as it appears and a parsed value, with the parsing rules stated. You then have something to audit, and disagreement between the two fields is a free error detector.

Failure: reading order in columns

A two-column academic paper, a newspaper page, or an invoice with a sidebar has no single correct linear order, and text extractors frequently interleave the columns line by line — producing sentences that alternate between two unrelated topics. The model then answers from a text that no human ever wrote. Rendering the page as an image avoids this entirely, which is one of the strongest arguments for the pixel path even when a text layer exists.

The same failure has a quieter cousin: headers, footers, page numbers and watermarks are interleaved into the text at page boundaries, so a sentence that runs across pages is interrupted by the document title and a date. It rarely produces a visibly wrong answer, which is why it goes unnoticed for a long time, and it steadily degrades anything built on top — chunk boundaries land in the wrong place, embeddings absorb boilerplate, and a retrieval system starts returning the page whose footer matched. Strip repeated headers and footers at ingestion by finding the lines that recur on most pages; it is fifteen lines of code and it improves everything downstream of it.

What to do instead

• Schema first, prose never. Define the output structure before choosing a model. Most of the failures above are schema failures wearing a vision costume.
• Send both representations. Extracted text plus the page image in one request. Cheap, and it collapses most transcription errors.
• Validate with arithmetic you already know. Line items should sum to the subtotal; subtotal plus tax should equal the total; a percentage column should approach 100. These checks catch silent misreads that no formatting rule will.
• Measure structure, not just text. If table extraction is the product, the metric used in the literature is TEDS (tree-edit-distance-based similarity), which scores the cell structure rather than the string — a transcription that is 99 % correct on characters and wrong on one colspan is not a good extraction.