HTML Extraction and Boilerplate Removal

A saved web page is mostly not the page. Extracting the article is the difference between a retrieval corpus where every document shares a thousand identical tokens of navigation and one where they do not — and shared tokens are exactly what breaks similarity.

Three things that are not the article

Boilerplate is not one problem. It is three, and they need different handling:

• Chrome — navigation, header, footer, sidebar, cookie banner, newsletter interstitial, “related articles”. Identical across every page on the site, which is precisely what makes it detectable and what makes it poisonous to retrieval.
• Invisible payload — <script> contents, inline CSS, SVG path data, base64 data URIs, and the JSON blob that modern frameworks embed to hydrate the client. That last one is the big one: a page rendered by a JavaScript framework often carries its entire content a second time as serialised state, so naive text extraction gives you the article twice.
• Markup overhead — the tags themselves. Even after removing chrome, keeping raw HTML rather than text means paying for every class attribute and every wrapper div.

Why get_text() is the wrong default

The three-line version everyone writes first is BeautifulSoup(html).get_text(). It has two specific problems beyond keeping the chrome.

First, it includes <script> and <style> contents unless you strip those elements yourself, because their children are text nodes as far as the tree is concerned. Second, it concatenates without separators, so a list of links becomes one long run-on word and <p>one</p><p>two</p> becomes onetwo — which corrupts sentence segmentation for everything downstream. The minimum defensible version:

from bs4 import BeautifulSoup

def crude_text(html: str) -> str:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup(["script", "style", "noscript", "template", "svg"]):
        tag.decompose()
    return soup.get_text(separator="\n", strip=True)

Use lxml rather than Python’s built-in html.parser for real-world pages: the built-in parser is stricter about malformed markup and will silently truncate a tree at the first unclosed tag it cannot recover from, which on a broken page costs you the rest of the article.

There is a third problem that only appears once you scrape modern sites. A server-rendered application embeds the state it needs to hydrate the client — in a <script> element with a type the browser will not execute, or as a JSON assignment to a global. That blob frequently contains the article body again, plus every other article in the same list, plus the entire navigation tree. Stripping <script> removes it, which is fine; not stripping it means your “article” is the article, a copy of the article, and nine summaries of unrelated articles, all of which will be chunked and embedded. It is the largest single source of duplicate content in a web corpus and it is invisible in a browser.

The extractors and what they key on

Purpose-built extractors do better than any tag list because they use signals a tag list cannot see. Knowing which signal each uses tells you when it will be wrong.

The structural signals are worth trying before any heuristic, because when they are present they are unambiguous: an <article> element, a role="main" landmark, an og:description, or a schema.org/NewsArticle JSON-LD block with an articleBody field. That last one hands you the article text with no heuristics at all, and a surprising number of publishing platforms emit it.

import json
from bs4 import BeautifulSoup
import trafilatura

def extract(html: str) -> str | None:
    # 1. Structured data, when the publisher gave it to us.
    soup = BeautifulSoup(html, "lxml")
    for node in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(node.string or "")
        except (json.JSONDecodeError, TypeError):
            continue
        for obj in (data if isinstance(data, list) else [data]):
            if isinstance(obj, dict) and obj.get("articleBody"):
                return obj["articleBody"]

    # 2. Heuristics, precision-biased: better a short article than a
    #    long one with the sidebar welded on.
    return trafilatura.extract(html, include_comments=False,
                               include_tables=True, favor_precision=True)

What removal is worth, derived

No measurement is needed to bound this, only the arithmetic — and the inputs are things you can read off your own corpus in a minute.

Let A be the tokens of actual article text on a page and B the tokens of everything else that survives naive extraction. Two costs follow. At index time, embedding costs are proportional to A + B rather than A, so the embedding bill is inflated by a factor of (A + B) / A. At query time, if you retrieve k chunks per request and N requests per month, the extra input tokens are N × k × chunk_size × B / (A + B), because the boilerplate is spread through the chunks rather than sitting in one of them.

Assume — and these are assumptions, to be replaced with your own numbers — a page where A = 900 and B = 2,100. The ratio (A + B) / A is 3.3, so indexing costs 3.3× what it needs to. With 100,000 pages at 3,000 tokens each and an embedding price of p per million tokens, that is 300M tokens rather than 90M — an avoidable 210M × p. At query time, roughly 70% of every retrieved chunk is furniture, which means to get k chunks’ worth of real content you retrieve and pay for about k / 0.3.

The second effect is worse than the first and it is not financial: when 70% of every chunk is the same text, embeddings of different pages move towards each other, and the retrieval quality problem that produces looks exactly like a chunking problem. It is not one.

Measuring it on your own pages

import statistics, tiktoken

enc = tiktoken.get_encoding("cl100k_base")   # any tokenizer; be consistent

def overhead(pages: list[str]) -> None:
    ratios = []
    for html in pages:
        raw = len(enc.encode(crude_text(html)))
        art = len(enc.encode(extract(html) or ""))
        if art:
            ratios.append(raw / art)
    ratios.sort()
    print("median", statistics.median(ratios),
          "p90", ratios[int(len(ratios) * 0.9)])

Report the median and the p90, not the mean: one page with a huge embedded JSON state blob will drag a mean past anything typical. The p90 is the number that tells you whether a subset of your sources needs per-site selectors.

Keeping the structure you need

Flattening to plain text throws away two things worth keeping. Headings give you a section path for each chunk — “Pricing > Enterprise > Support SLAs” — which is among the most useful metadata you can attach. And tables become unreadable as prose: a five-column table flattened to text loses the column association entirely, so a row reads as seven unrelated numbers.

Markdown output is the usual compromise, and both trafilatura (with output_format="markdown") and the various HTML-to-Markdown converters will give it to you. It keeps headings, lists and table structure at a cost of a few tokens of punctuation per element — far less than the HTML tags it replaces, and far more useful than flat text to a model that has seen a great deal of Markdown.

Two things are worth keeping outside the text entirely. Link targets: a document full of inline URLs tokenizes badly and adds nothing to retrieval, so strip the href into a side table keyed by document if you need it and drop it otherwise. And image alt text: keep it, because on a well-built page it is a description of a figure that the text refers to, and losing it turns “as shown below” into a dangling reference.

Whichever extractor you settle on, pin its version and record it per document, exactly as with a PDF parser. Extractors change their heuristics between releases, and an upgrade that improves the average page can quietly change what is included on a whole class of sources. With the version recorded, that is a targeted re-extraction; without it, it is an unexplained drift in retrieval quality with no way back to the cause.