Reading PDFs, DOCX and HTML Into Clean Text

12 min read · updated August 4, 2026

Every document pipeline is three steps: work out what the file is, pull text out of it, and clean the text. The middle step has a different library per format and each of those libraries fails quietly in its own particular way — which is the part worth writing down.

The shape: detect, extract, normalise

Do not trust the file extension. A .pdf that is actually HTML, or a .docx that is a legacy .doc, is common enough in real uploads that sniffing the magic bytes is worth the eight lines.

# detect.py
from pathlib import Path


def sniff(path: Path) -> str:
    head = path.open("rb").read(8)
    if head.startswith(b"%PDF-"):
        return "pdf"
    if head.startswith(b"PK\x03\x04"):
        return "zip"          # docx, xlsx, pptx, odt, epub — all zip containers
    if head.startswith(b"\xd0\xcf\x11\xe0"):
        return "ole2"         # legacy .doc/.xls — needs conversion, not parsing
    sample = head + path.open("rb").read(2048)
    lowered = sample.lower()
    if b"<html" in lowered or b"<!doctype html" in lowered:
        return "html"
    return "text"

The ole2 case is the one to handle explicitly rather than let fail. A legacy .doc cannot be read by python-docx at all; the practical route is converting it first with LibreOffice in headless mode (soffice --headless --convert-to docx) and then parsing the result.

PDF

pip install pypdf. The name matters: the library was called PyPDF2 until it was renamed to pypdf, and the extraction method was extractText() before version 2 and extract_text() after. A tutorial using the old spelling is pre-2022 and probably wrong about other things too.

# extract_pdf.py
from pypdf import PdfReader


def pdf_to_text(path: str) -> list[str]:
    """Return one string per page. Empty strings are meaningful — see below."""
    reader = PdfReader(path)
    if reader.is_encrypted:
        reader.decrypt("")          # many PDFs are encrypted with an empty owner password
    return [(page.extract_text() or "") for page in reader.pages]


def looks_scanned(pages: list[str], *, min_chars: int = 50) -> bool:
    """True if most pages produced almost no text: an image-only PDF."""
    if not pages:
        return True
    empty = sum(1 for page in pages if len(page.strip()) < min_chars)
    return empty > len(pages) * 0.6

What this does badly, and what to do about each:

Scanned pages produce nothing. A PDF of photographs has no text layer, and extract_text() returns "" without error. This is the single most common silent failure in document pipelines: the file parsed, the pipeline succeeded, and the answer is based on nothing. looks_scanned above is the check; the remedy is OCR, which is a different pipeline — the OCR pipeline and vision models versus dedicated OCR cover the options.
Multi-column layouts interleave. Extraction follows the order objects were drawn, not reading order, so a two-column academic paper can come out with lines alternating between columns. This corrupts meaning without looking corrupt. pdfplumber gives you word positions, so you can sort by x-coordinate and split columns yourself.
Tables become word soup. Cell boundaries are visual, not structural. If tables carry the meaning — invoices, financial statements — use pdfplumber’s page.extract_tables(), which returns lists of rows, and render them as Markdown or CSV before handing them to a model.
Headers, footers and page numbers repeat. On a 200-page document that is 200 copies of the same line in your embeddings. Detect them by finding short lines that appear on more than half the pages and drop them.
Ligatures and hyphenation. Expect ﬁ and ﬂ as single characters, and words split across lines as exam-\nple. Both are handled in the normalisation step below.

# tables, when they matter
import pdfplumber

def pdf_tables(path: str) -> list[list[list[str | None]]]:
    out = []
    with pdfplumber.open(path) as pdf:
        for page in pdf.pages:
            out.extend(page.extract_tables())
    return out

DOCX

pip install python-docx — note that the package name is python-docx and the import is docx, which is a long-standing source of “module not found” confusion.

# extract_docx.py
from docx import Document


def docx_to_text(path: str) -> str:
    document = Document(path)
    blocks: list[str] = []

    for paragraph in document.paragraphs:
        text = paragraph.text.strip()
        if not text:
            continue
        style = (paragraph.style.name or "").lower()
        if style.startswith("heading"):
            blocks.append(f"\n## {text}\n")     # keep the structure a model can use
        else:
            blocks.append(text)

    for table in document.tables:
        rows = ["| " + " | ".join(cell.text.strip() for cell in row.cells) + " |"
                for row in table.rows]
        if rows:
            blocks.append("\n".join(rows))

    return "\n\n".join(blocks)

document.paragraphs skips text in tables. They are separate collections, which is why the function above walks both. Missing this loses whole documents that happen to be laid out as a table.
Order between the two is not preserved. The code above appends all paragraphs, then all tables. For documents where interleaving matters, iterate the underlying body elements in document order instead — that means working with the XML, and it is worth doing only when it matters.
Headers, footers, footnotes and comments are elsewhere. They live on section and part objects, not in paragraphs. Usually you want them excluded; know that you excluded them.
Tracked changes are included as accepted text. A document with unresolved edits extracts as though every change was approved, which for contracts is a real correctness problem.
.doc is not .docx. python-docx raises on the old binary format. Convert first.

HTML

Two different jobs get confused here. Getting all the text out of an HTML file is one thing; getting the article out of a web page full of navigation, cookie banners and related-links boxes is another, and for anything crawled you want the second.

# extract_html.py
from bs4 import BeautifulSoup

DROP = ["script", "style", "nav", "header", "footer", "aside",
        "noscript", "form", "svg", "template"]


def html_to_text(html: str) -> str:
    soup = BeautifulSoup(html, "lxml")      # or "html.parser" with no extra install
    for tag in soup(DROP):
        tag.decompose()
    text = soup.get_text(separator="\n")
    lines = [line.strip() for line in text.splitlines()]
    return "\n".join(line for line in lines if line)

For real pages, a boilerplate remover does far better than a tag blacklist. trafilatura is the usual choice — trafilatura.extract(html) returns the main content as text, or None when it decides there is no article, and that None is information rather than a failure.

Client-rendered pages have no content in the HTML. A fetch returns the shell and your extractor returns forty characters of noise. Detect it — very short output from a large HTML file — and either skip the page or render it with a headless browser.
get_text() with no separator glues words together. <p>one</p><p>two</p> becomes onetwo. Always pass a separator.
Parser choice changes the output on broken markup. lxml is fast and forgiving; html.parser needs no compiled dependency; html5lib is slowest and most faithful to what a browser would do. Pick one and pin it, because switching silently changes your extracted text.
Encoding lies. The charset in a Content-Type header is frequently wrong. Decode with errors="replace" so a single bad byte cannot kill a batch, and count the replacement characters as a quality signal.

The normalisation nobody writes

The same cleanup applies to all three formats and it is what makes chunking and embedding behave. It is fifteen lines and it removes a whole class of retrieval bugs.

# normalise.py
import re
import unicodedata

DEHYPHENATE = re.compile(r"(\w+)-\n(\w+)")
MANY_NEWLINES = re.compile(r"\n{3,}")
SPACES = re.compile(r"[ \t]{2,}")


def normalise(text: str) -> str:
    # NFKC folds ligatures (ﬁ -> fi) and full-width forms into plain ASCII forms
    text = unicodedata.normalize("NFKC", text)
    text = text.replace("\r\n", "\n").replace("\r", "\n")
    text = text.replace("\u00a0", " ")           # non-breaking space
    text = text.replace("\u200b", "")            # zero-width space
    text = DEHYPHENATE.sub(r"\1\2", text)        # exam-\nple -> example
    text = SPACES.sub(" ", text)
    text = MANY_NEWLINES.sub("\n\n", text)
    return text.strip()

NFKC is the one doing the heavy lifting. Without it, a document using the ﬁ ligature contains a token your query will never match, and neither keyword search nor an embedding will connect “classiﬁcation” with “classification”. The zero-width space removal matters for anything copied out of a web page.

The dehyphenation rule is the one to watch. It joins across a line break, which is right for PDF wrapping and wrong for a genuine end-of-line hyphen in “well-\nknown”. If your corpus is hyphen-heavy, check the joined word against a dictionary before accepting the join.

Telling a good extraction from a bad one

Every batch pipeline needs a rejection rule, or bad extractions flow silently into an index and degrade every answer that touches them. These four checks are cheap and catch most of it:

Characters per page. Under about 50 on most pages means an image-only PDF. Route to OCR or to a human, do not index.
Replacement-character ratio. More than a fraction of a per cent of � means the encoding was guessed wrong. Re-decode before indexing.
Mean word length. Below about 2 or above about 15 suggests the text is spaced-out gibberish or has lost its spaces entirely — both are classic broken-extraction signatures.
Repeated-line ratio. If more than a third of lines are duplicates, you have headers and footers, not content. Strip them and re-check.

Record the result of each check alongside the document, so a bad answer can be traced back to a bad extraction in one query instead of a re-run. Document ingestion covers the pipeline this feeds, and chunking strategies the step immediately after.