Extracting Data From a Document With a Sidebar and Main Column

A magazine-style page, an annual report or a product datasheet drops a quoted sentence in large type across the middle of a column, or a shaded box of caveats down the side. Extract it naively and that text arrives in the middle of an unrelated paragraph — and if it is a pull quote, it arrives twice.

What the splice does to the text

The damage is worse than untidy output, because it is invisible downstream. A sentence spliced into the middle of a paragraph changes the meaning of the chunk it lands in, and if that chunk is embedded for retrieval, the embedding is of a sentence that does not exist in the document. A duplicated pull quote gives you two chunks with nearly identical vectors, which distorts any deduplication or clustering you do later.

If a model is doing the extraction, the failure is quieter still: the model reads the spliced text as though it were continuous and produces a fluent, confident field value derived partly from a caption. Nothing in the output signals that this happened, which is a good reason to fix the layout before the model sees it rather than asking the model to cope.

Three things that look like a sidebar

They need different handling, and conflating them is the usual reason a fix half-works.

• A pull quote. A sentence taken verbatim from the body and reset in large type as a visual break. It contains no new information. It must be removed from the linear text, or it appears twice.
• A call-out box. A shaded or ruled panel with unique content: a worked example, a warning, a case study, a definition. It must be kept, labelled, and placed somewhere sensible. Dropping it is a silent data loss and it is the more expensive mistake of the two.
• A margin note. Short text in the outer margin, aligned with a specific paragraph. In legal and technical documents these carry section numbers, clause references or revision marks, and they are load-bearing: the margin note is often the only place the clause identifier appears.

Separating by geometry

Start from word bounding boxes and from the page’s vector graphics, which most PDF libraries expose separately from text. A call-out box usually has a filled rectangle or a ruled border behind it, and that rectangle is in the file. Any word whose box is contained within a filled rectangle belongs to that panel, which is an exact test rather than a heuristic, and it survives layouts where the panel is not rectangular in the column sense — a panel straddling both columns at the foot of the page, for instance.

Where there is no drawn rectangle, fall back on the same projection-profile reasoning used for two-column reading order, with one addition: a sidebar is a region whose x-extent is stable down a long stretch of the page and much narrower than the main text block, and whose line spacing frequently differs from the body. Detect it as a band, not as a column, and do not assume it starts at the top of the page — sidebars often begin partway down.

Typographic signals

Geometry alone will not tell you what a region is, and character-level font metadata will. PDF text extraction libraries expose per-character font name and size, so you can compute the body’s modal font and size and then flag regions that deviate.

import pdfplumber
from collections import Counter

page = pdfplumber.open("report.pdf").pages[6]
chars = page.chars

body = Counter((c["fontname"], round(c["size"], 1)) for c in chars).most_common(1)[0][0]

def looks_like_display(word_chars):
    fonts = {(c["fontname"], round(c["size"], 1)) for c in word_chars}
    return all(f != body for f in fonts) and max(
        c["size"] for c in word_chars
    ) > body[1] * 1.25

The 1.25 multiplier is a starting threshold, not a constant of nature; pick it by looking at the size histogram of a handful of pages from the template you are processing. What matters is the shape of the reasoning: the body text is the mode, and a pull quote is set conspicuously larger by design, because its entire purpose is to be conspicuous.

Reversed text — white on a dark panel — is worth a particular note. In a digital PDF it extracts normally and the fill colour is available as metadata, so it is detectable. In a scan, a binarisation step tuned for dark-on-light can erase it completely, and you will never know the panel existed. If your corpus contains reversed panels, check for large dark regions before binarising.

Deduplicating the pull quote

Once you have candidate display regions, decide whether each duplicates body text. Normalise both sides hard before comparing: collapse whitespace, fold case, strip the decorative quotation marks that pull quotes are usually wrapped in, remove the ellipsis that marks an elision, and normalise the ligatures and the various dash and quote code points.

Then look for the normalised quote as a substring of the normalised body of the same page or its neighbour. Exact containment catches most of them. A pull quote that has been abridged with an ellipsis will not be contained, so fall back to a token-overlap ratio against the body and treat a high overlap over a short span as a duplicate.

Never delete on a near match alone. Mark the region as pull_quote with the offset of the body text it matched, exclude it from the linear reading text, and keep it in the structured output. That way a wrong decision is inspectable rather than a hole, which is the same principle behind keeping the original label in the field-level audit trail.

Where the sidebar goes in the output

A call-out box has to be linearised eventually, and there is no canonical answer for where. Three rules make the result usable.

• Anchor it to the paragraph it overlaps. Emit the panel immediately after the body paragraph whose vertical extent overlaps the panel’s top edge. That approximates what a reader does and keeps it near its context.
• Keep it as its own block, with a type. Do not concatenate it into surrounding prose. A downstream chunker for retrieval should be able to keep a panel whole, and a schema field that should only be populated from the main narrative should be able to exclude panels.
• Carry the coordinates. Every block keeps its page number and bounding box, so that a reviewer can be shown the exact region on the exact page. Without that, a field extracted from a sidebar is unverifiable, and source highlighting in the review queue has nothing to point at.