Extracting Text From a Two-Column Academic PDF in Reading Order

You extract a paper and get sentences that alternate between two unrelated thoughts, a paragraph that ends mid-clause and resumes four lines later, and references interleaved with the conclusion. Nothing is missing. Everything is in the wrong order, because the extractor walked the page left to right across both columns.

Sentences that alternate

The tell is that the output is coherent when you read every other line. If you can reconstruct the text by taking alternate lines, the extractor is emitting line by line in visual top-to-bottom order across the full page width, so each output line is the left column’s line followed by the right column’s line at the same height.

A second and subtler variant produces text that is in order within long stretches and then jumps. That happens when the extractor sorts by the order operators appear in the content stream rather than by position, and the generator happened to emit one column at a time for most of the page.

Why a PDF has no reading order

A PDF page is a sequence of drawing operators. Text is placed with positioning operators that set a matrix and then show a string; the format has no requirement that those operators appear in the order a human reads, and no structure that says “this is a column”. A typesetter is free to draw all the body text of one column, then the other, then the headers, then the footnotes, and many do exactly that for font-switching efficiency.

So there are two orders in a PDF and neither is reading order: the order the operators appear in the file, and the order the resulting glyphs appear on the page geometrically. Extractors differ in which one they use, and libraries expose the choice. Poppler’s pdftotext has a raw mode that emits content-stream order and a layout mode that tries to preserve the physical arrangement; PyMuPDF’s block extraction takes a sort flag that orders blocks top-to-bottom then left-to-right. Neither of those is column detection, which is why turning on the sort flag frequently makes the interleaving worse rather than better: it enforces exactly the wrong order more consistently.

The general problem this belongs to is document layout analysis, and the classical answer — still the right answer for a clean digital two-column paper — is a recursive projection-profile cut, usually called XY-cut. It requires no model and no training data.

Finding the gutter

Extract words with their bounding boxes rather than lines of text. Then build a vertical projection profile: for each x position across the page, count how many word boxes cover it. A two-column layout produces two humps separated by a run of x positions where the count is zero. That run is the gutter, and its midpoint is your cut.

import pdfplumber

page = pdfplumber.open("paper.pdf").pages[3]
words = page.extract_words()

# Occupancy per horizontal position, at 1-point resolution.
width = int(page.width) + 1
cover = [0] * width
for w in words:
    for x in range(int(w["x0"]), min(int(w["x1"]) + 1, width)):
        cover[x] += 1

# Longest interior run of zero coverage = the gutter.
runs, start = [], None
for x in range(width):
    if cover[x] == 0 and start is None:
        start = x
    elif cover[x] != 0 and start is not None:
        runs.append((start, x))
        start = None
interior = [r for r in runs if r[0] > 0.15 * width and r[1] < 0.85 * width]
gutter = max(interior, key=lambda r: r[1] - r[0])
cut = (gutter[0] + gutter[1]) / 2

Two guards matter. Exclude the page margins, or the widest zero run is the margin and the cut lands off the text. And require the gutter to be wide enough to be real: a run of two or three points is inter-word space that happened to line up vertically, whereas a genuine column gutter in a two-column paper is a substantial fraction of the column width. If no run clears the threshold, the page is single-column and you should not cut it at all.

Band first, then cut

One global gutter is the mistake that survives longest, because it works on the middle pages of a paper and fails on the first one. An academic paper is not two columns; it is a sequence of horizontal bands that are each either full width or two columns. The title, the author block, the abstract on many templates, a wide figure, a wide table and the copyright footer are all full width, and a global cut slices them in half.

So invert the order of operations. Cut horizontally first, into bands separated by runs of empty y positions, then test each band independently for a vertical gutter, then recurse into the halves of any band that has one. That recursion is what makes the classical algorithm work on real papers, and it terminates naturally when a region has no qualifying gap in either direction.

1. Compute the horizontal projection profile and split the page at empty y-runs wider than the body line spacing, producing bands.
2. For each band, compute a vertical profile and look for an interior zero run wider than your gutter threshold.
3. If one exists, split the band and recurse into each half. If not, emit the band’s words sorted top-to-bottom, then left-to-right within a line.
4. Concatenate bands in y order, and within a split band emit the left region completely before the right one.

Footnotes are the case worth special handling. They sit below a short horizontal rule at the bottom of a column, in a smaller size, and they are part of neither the paragraph above them nor the next column. Look for a line segment much shorter than the column width with small text beneath it and emit that region separately, labelled, rather than inline — otherwise a footnote lands in the middle of a sentence in your output.

Rejoining text across the break

Getting the order right is half of it. The other half is that a paragraph broken across a column boundary has to be stitched back together without introducing artefacts.

• Do not insert a space at the join by default. If the left column ends mid-word, the word is split across the columns and a space corrupts it. Join with a space only when the preceding character is not a hyphen and the following text starts a new sentence or the preceding line ended at the right margin with a complete word.
• Handle the hyphen carefully. A trailing hyphen at a line or column break is usually a soft hyphen inserted by the typesetter and should be removed when rejoining. But some are real: self- at a line end followed by organising should rejoin as self-organising. A dictionary check on both the joined and the hyphenated form is the pragmatic resolution, and where neither wins, keep the hyphen — a wrong hyphen is visible to a reader, a wrongly deleted one is not.
• Normalise ligatures. Text set in a typical academic font contains the single code points for fi and fl rather than two letters, which breaks search and any downstream string matching. Apply a compatibility normalisation after extraction, not before, so the coordinate work is unaffected.

When the file already knows

Some PDFs carry a logical structure tree that states the reading order explicitly — a tagged PDF, which is what the accessibility requirements in the PDF/UA family exist to produce, published by ISO. When one is present, use it: it is the author’s own statement of the order and it beats any geometric heuristic. The catch is that scientific PDFs produced by traditional TeX toolchains are frequently untagged, and a file can also carry a structure tree that is present but wrong, having been generated by a tool that guessed.

So the right control flow is: use the structure tree if it exists and is coherent, verify it by checking that the tagged order is consistent with the geometry, and fall back to band-then-cut when it is not. If you are recovering text from scans rather than digital files, none of this applies until after recognition — see the OCR pipeline and, for layouts with boxed asides rather than plain columns, separating a sidebar from the main column.