Web Scraping for AI: The Legal and Technical Constraints

The technical constraints on crawling are written down and easy to honour. The legal ones are not one thing at all — they are three separate regimes that people routinely collapse into a single question, and the collapse is where the trouble starts.

robots.txt is a standard now

For twenty-five years the Robots Exclusion Protocol was a convention with an informal spec. Since 2022 it is RFC 9309, which pins down the parts implementations used to disagree about. Three of those matter in practice.

• The most specific rule wins, not the first one. A Disallow: /docs/ and an Allow: /docs/public/ are resolved by path length, so the longer Allow wins for everything under it. Implementations that took the first matching line got this backwards.
• One group applies. The crawler picks the group whose User-agent best matches its own token and ignores every other group including *. Being named in the file means the wildcard rules do not apply to you at all — which cuts both ways.
• Status codes have meanings. A 404 means everything is allowed. A 5xx should be treated as fully disallowed, at least temporarily. Caching for up to 24 hours is explicitly sanctioned, so you do not have to fetch it per request.

Python’s standard library has a parser, and it is worth knowing that Crawl-delay is not in RFC 9309 — it is a widely honoured extension, and urllib.robotparser exposes it anyway via crawl_delay(), returning None when absent.

from urllib.robotparser import RobotFileParser
from urllib.parse import urlsplit

class Robots:
    def __init__(self, agent: str, default_delay: float = 1.0):
        self.agent, self.default_delay, self.cache = agent, default_delay, {}

    def _for(self, url: str) -> RobotFileParser:
        origin = "{0}://{1}".format(*urlsplit(url)[:2])
        if origin not in self.cache:
            rp = RobotFileParser()
            rp.set_url(origin + "/robots.txt")
            try:
                rp.read()               # 404 -> allow all, per RFC 9309
            except Exception:
                rp.disallow_all = True  # 5xx or unreachable -> stay out
            self.cache[origin] = rp
        return self.cache[origin]

    def allowed(self, url: str) -> bool:
        return self._for(url).can_fetch(self.agent, url)

    def delay(self, url: str) -> float:
        d = self._for(url).crawl_delay(self.agent)
        return float(d) if d is not None else self.default_delay

Also read sitemap.xml, which RFC 9309 keeps as a robots.txt directive. It gives you the publisher’s own list of canonical URLs with lastmod dates — cheaper and more accurate than discovering pages by following links, and the lastmod field feeds directly into deciding what to re-fetch.

The AI-specific tokens

Since 2023 the major AI vendors have published distinct user-agent tokens so that publishers can allow ordinary search indexing while refusing AI training or retrieval. GPTBot, Google-Extended, ClaudeBot, CCBot (Common Crawl) and PerplexityBot are among the names that now appear in robots.txt files, and several of them separate crawling for training from fetching a page live to answer a user’s question. A site that disallows one and allows another has expressed a preference that a single blanket rule cannot represent.

Two consequences for you. First, if you are crawling on your own behalf, pick a distinctive token, put a URL in your user-agent string that explains who you are, and honour rules addressed to it — an anonymous python-requests/2.31.0 is what gets an IP range blocked. Second, if you are consuming somebody else’s crawl (a public dataset, a commercial API), the exclusions that applied at crawl time are the ones baked into the data, and they are not yours to re-derive. Record the provenance so you can answer the question later.

Where terms of service take over

robots.txt is a machine-readable request. It is not the only constraint and in most cases it is not the binding one. There are three separate regimes and they resolve independently:

The practical middle ground most teams land on: prefer official APIs and bulk feeds where they exist; honour robots.txt strictly; keep per-host concurrency low and identify yourself; store the source URL and fetch timestamp on every document so provenance is answerable; and get a lawyer to look at anything that is going into a product, because none of the above is legal advice and the answer genuinely depends on jurisdiction and use.

A crawler that does not get blocked

Politeness is mostly per-host concurrency, and the mistake is applying a global rate limit to a crawl that is 90% one host. Key the limiter on the host:

import asyncio, time, httpx
from collections import defaultdict
from urllib.parse import urlsplit

UA = "AcmeDocsBot/1.0 (+https://acme.example/bot)"

class HostLimiter:
    def __init__(self):
        self.locks, self.next_ok = defaultdict(asyncio.Lock), defaultdict(float)

    async def wait(self, host: str, delay: float):
        async with self.locks[host]:
            gap = self.next_ok[host] - time.monotonic()
            if gap > 0:
                await asyncio.sleep(gap)
            self.next_ok[host] = time.monotonic() + delay

async def fetch(client, url, robots, limiter):
    if not robots.allowed(url):
        return None
    host = urlsplit(url).netloc
    await limiter.wait(host, robots.delay(url))
    r = await client.get(url, headers={"User-Agent": UA},
                         follow_redirects=True, timeout=30)
    if r.status_code == 429 or r.status_code >= 500:
        # Retry-After may be seconds or an HTTP-date; seconds is the common case
        wait = float(r.headers.get("Retry-After", 60))
        await asyncio.sleep(min(wait, 300))
        return None
    return r

A 429 is information, not an error to retry through. Treat a host that returns them as one you are crawling too fast, halve its rate for the rest of the run, and respect Retry-After when it is present instead of backing off on your own schedule.

Fetching the same page twice, cheaply

A corpus that refreshes is mostly re-fetching things that have not changed. HTTP has had the answer since 1999: store the ETag and Last-Modified headers with each document, send them back as If-None-Match and If-Modified-Since, and a well-behaved server answers 304 Not Modified with no body at all.

headers = {"User-Agent": UA}
if row.etag:
    headers["If-None-Match"] = row.etag
if row.last_modified:
    headers["If-Modified-Since"] = row.last_modified

r = await client.get(url, headers=headers)
if r.status_code == 304:
    touch(row, checked_at=now())        # still current, nothing re-parsed
elif r.status_code == 200:
    if sha256(r.content) != row.raw_sha256:
        enqueue_reprocess(url, r.content, r.headers)
    save_validators(row, r.headers.get("ETag"),
                    r.headers.get("Last-Modified"))

Keep the content hash check even when the validators say the page changed: plenty of sites emit a fresh ETag on every render, or change a timestamp in the footer, and the hash of the extracted article rather than the raw HTML is the signal you actually want. Comparing extracted text is what stops a rotating advertisement from triggering a re-embedding of the whole page.

Two more headers pay for themselves on a crawl of any size. Send Accept-Encoding: gzip, br and let the server compress: HTML compresses several-fold, and the saving is on the bytes you are charged to receive as well as on the time. And cap the response size before you read the body — a streaming read with a byte budget, rather than a call that returns the whole thing — because somewhere in any crawl is a URL that serves a multi-gigabyte file with an innocuous content type, and discovering it by exhausting a worker’s memory is an avoidable way to spend an afternoon.

Finally, record the response status and the final URL after redirects on every fetch, not just on the successes. A source that starts answering 403 to your crawler stops updating silently — the last good copy stays in the index, looking exactly like a document that has not changed. The distinction between “unchanged” and “unreachable since March” only exists if you wrote it down.