PDF Parsing: Why It's Still Hard

PDF extraction is not hard because the parsers are bad. It is hard because the thing you want — text, in order, in paragraphs — is not what the file contains, and in many documents it cannot be recovered exactly.

What is actually in the file

A PDF page’s content stream is a sequence of drawing operators, specified in ISO 32000-1. The text ones are small in number: BT begins a text object, Tf selects a font and size, Td and Tm position the text matrix, Tj and TJ show a string, ET ends the object. A line of a document might be emitted like this:

BT
  /F1 11 Tf
  72 720 Td
  [(Inv) 20 (oice) -250 (T) 80 (otal:)] TJ
  180 0 Td
  (1,240.00) Tj
ET

Read that carefully, because everything about PDF extraction follows from it. There is no word “Invoice” in the file — there is Inv, then a kerning adjustment of 20 units, then oice. The numbers inside the TJ array move the pen backwards in thousandths of an em to tighten letter pairs. A parser must decide, for each of those adjustments, whether it is kerning inside a word or a space between words, and the file does not say. That decision is a threshold, and thresholds have failure cases in both directions: “Inv oice” when it is too low, “InvoiceTotal” when it is too high.

Nor is there a space character between “Total:” and “1,240.00”. There is a Td that moves the pen 180 units to the right. Whether that gap is a single space, a tab stop or the boundary between two table columns is an inference from geometry.

The ToUnicode problem

The bytes in a Tj string are not Unicode. They are character codes to be looked up in the selected font’s encoding, which for an embedded subset font is whatever the producing application chose — frequently a dense numbering of only the glyphs the document uses, so code 1 is whichever letter happened to appear first.

A PDF may carry a /ToUnicode CMap that maps those codes back to Unicode. It is optional. When it is missing — common in output from older TeX toolchains, from some CAD and reporting tools, and from files that have been through a re-writer — extraction has no way to recover text. pdfminer.six is explicit about this and emits (cid:34)-style markers for codes it cannot map; other libraries may return the raw code points, which decode to plausible-looking nonsense that no assertion on “is this text?” will catch unless you check the letter distribution.

Ligatures are the same problem in miniature. A well-formed /ToUnicode maps the fi ligature glyph to U+FB01, which is a real character that your tokenizer, your search index and your dedupe hash will all treat as different from “fi”. Normalising with unicodedata.normalize("NFKC", text) decomposes it back to two ASCII letters, which is why the canonical form belongs immediately after extraction rather than at query time.

There is a diagnostic worth building for this, because the symptom is so easy to miss. Take the extracted text of a page, count how many of its characters fall outside the set your language actually uses, and compare that to the fraction of characters that are spaces. A page with a broken font map typically has a normal-looking space ratio — the positioning was fine, only the code-to-character map was not — and an abnormal distribution of letters, with runs that never appear in real words. That combination is a much better detector than any check on length, and it is the only one that catches the case where the missing map produced letters rather than cid markers.

Reading order is not stored

Content streams are emitted in whatever order the producer drew them. For a single-column report that is usually top to bottom. For a two-column paper, a magazine layout, or anything with pull quotes and sidebars, it is frequently not, and a naive extraction interleaves the columns line by line — producing text that is locally readable and globally meaningless, which is the worst outcome because nothing detects it.

Tagged PDFs (PDF/UA, and the structure tree from ISO 32000 section 14.7) do carry a logical order, and when a document has one it is authoritative. Most documents do not. Everything else is geometric reconstruction: cluster the text spans into blocks by position, order the blocks, order lines within blocks. PyMuPDF exposes exactly this via page.get_text("dict"), which returns blocks → lines → spans with a bounding box on each, and a sort=True flag on the plain-text call that orders by position instead of by drawing order.

import fitz            # PyMuPDF

def two_column_aware(page, gutter_ratio=0.45):
    """Split blocks by which half of the page they start in, then read
    the left column top-to-bottom before the right one."""
    width = page.rect.width
    blocks = page.get_text("blocks")     # (x0,y0,x1,y1,text,bno,btype)
    left  = [b for b in blocks if b[0] < width * gutter_ratio]
    right = [b for b in blocks if b[0] >= width * gutter_ratio]
    order = sorted(left, key=lambda b: b[1]) + sorted(right, key=lambda b: b[1])
    return "\n".join(b[4] for b in order if b[6] == 0)   # 0 = text block

That heuristic is crude and it is also enough for a large class of academic PDFs. The point is that you are writing a layout heuristic whether you know it or not; writing it explicitly means you can tune it when it is wrong.

Four strategies and what each must fail on

These compose better than they compete. The pattern that survives contact with a mixed corpus is a cascade: try the text layer, run the extraction assertions from the ingestion checks, and fall through to OCR or a vision model only for the documents that fail them. That keeps the per-page cost near zero for the born-digital majority.

A bake-off you can run

Parser rankings published elsewhere are about somebody else’s documents. Build a gold set of twenty to fifty of your hard pages — one per failure mode you have actually seen — transcribe them by hand once, and score candidates against it:

import jiwer, json, pathlib

def score(parsers: dict, gold_dir="gold"):
    # gold/<name>.pdf next to gold/<name>.txt, transcribed by a human
    rows = []
    for txt in pathlib.Path(gold_dir).glob("*.txt"):
        pdf = txt.with_suffix(".pdf")
        want = jiwer.Compose([jiwer.ToLowerCase(),
                              jiwer.RemoveMultipleSpaces(),
                              jiwer.Strip()])(txt.read_text())
        for name, fn in parsers.items():
            got = want.__class__ and jiwer.Compose([
                jiwer.ToLowerCase(), jiwer.RemoveMultipleSpaces(), jiwer.Strip()
            ])(fn(pdf))
            rows.append({"doc": txt.stem, "parser": name,
                         "cer": jiwer.cer(want, got),
                         "wer": jiwer.wer(want, got)})
    print(json.dumps(rows, indent=1))

Character error rate is the metric to lead with, because word error rate punishes the space-insertion decisions from the first section twice — once as a deletion and once as an insertion — and a parser that gets every letter right and some spaces wrong is much more useful than that number suggests. Fifty hand-transcribed pages is a day of work and it replaces every argument about parsers with a table.