Paywalls, Crawlers and Selective Access

A paywall and a crawler want incompatible things: one hides the text, the other needs it to rank the page. There is exactly one documented, sanctioned way to resolve that — structured data declaring which part is paid — and everything else that serves a crawler more than a reader is the spam policy people mean by cloaking.

The problem in one sentence

If a crawler receives only your headline and a sign-up prompt, that is all it can index, so your article competes on two hundred characters against free articles with two thousand words. If instead you detect the crawler and serve it the full text, you are serving different content to a crawler than to a reader, which is the definition of the behaviour search operators penalise. The structured-data exception exists precisely to make the second thing legitimate when you declare it.

What cloaking is, and why this is not it

Cloaking, in the spam-policy sense, is presenting different content to search crawlers than to users in order to manipulate rankings. The deceptive intent is doing the work in that definition, and the treatment is severe: it is one of the few things that reliably produces a manual action rather than a gradual ranking effect.

The paywall case is carved out by declaration. When you mark the paywalled section in your structured data, you are telling the operator in machine-readable form: this part is behind payment, the crawler is seeing it, a logged-out reader is not, and here is exactly which part. The difference between the crawler view and the reader view is no longer hidden, and it is therefore no longer deception.

Three practices that are not covered and remain policy violations regardless of markup: serving different text to the crawler than sits behind the paywall; keyword-stuffing the crawler-visible version; and using the exception on content that is not actually paid. The exception describes an arrangement, and if the arrangement is not real, the declaration is a false statement.

The structured-data paywall exception

The mechanism uses two properties on the article: an isAccessibleForFree flag set to false, and a hasPart entry identifying the paywalled section by CSS selector. The selector must match the element wrapping the paid content on the actual page.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "NewsArticle",
  "headline": "The full headline as displayed",
  "datePublished": "2026-08-04",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".paywalled-body"
  }
}
</script>

<article>
  <h1>The full headline as displayed</h1>
  <p class="lede">The free lede paragraph, visible to everyone.</p>
  <div class="paywalled-body">
    <!-- Full text. Present in the HTML for crawlers; hidden or
         truncated for logged-out readers by the paywall logic. -->
  </div>
</article>

1. Wrap exactly the paid part in an element with a stable class, and make the selector match it. If the selector matches nothing, the declaration is inert and you are back to undeclared cloaking.
2. Use multiple hasPart entries if the page has several paid sections; the property accepts an array.
3. Set isAccessibleForFree to true on articles that are free. A site that marks everything as paid, including its free content, is making a false statement in the direction that happens to be checkable.
4. Keep the free portion substantive. Metered and lead-in models both require that a reader gets something real, and a page whose free portion is a headline and a wall converts badly for humans regardless of how it indexes.
5. Validate it alongside the rest of your markup — see structured data that machines read.

Access models being used now

The trade-off across the table is one axis: the more the crawler can see, the more anything else that fetches your page can see too. Client-side hiding in particular is not a security control — the text is in the response body, and curl reveals it. If your commercial position depends on the text not leaving your server, only server-side truncation delivers that, and it delivers it at the cost you would expect.

Treating AI crawlers differently from search

The common intent is: full text to search crawlers, nothing to bulk AI training crawlers, and a decision to be made about assistant fetchers that link back. The mechanics, in increasing order of reliability:

• robots.txt. Disallow the training tokens, allow the search tokens. Voluntary, and it stops nobody who declines — see robots.txt for AI crawlers.
• Edge rules on verified identity. Allow full text only to requests whose address is confirmed against a published crawler range or forward-confirmed reverse DNS, and truncate for everything else. This is enforcement rather than a request, and the verification method matters — two verification regimes covers which applies to whom.
• Authentication. The only complete answer, and the one that removes you from every index that will not log in.

There is an important asymmetry to be clear-eyed about. Serving full text to verified Googlebot and truncated text to everything else is a declared arrangement covered by the paywall markup. Serving full text to an unverified user-agent string is not a control at all, because the string is free to send: anybody who wants your full text sets their user agent to Googlebot and gets it. If you are gating on identity, gate on verified identity.

Charging for access at the protocol level

A newer model deserves its own note because it changes the shape of the question. Rather than allow or deny, the server answers a crawler with 402 Payment Required and a statement of terms, and a crawler willing to pay retries with proof. Cloudflare launched a pay-per-crawl mechanism along these lines in beta in 2025, alongside blocking AI crawlers by default for new domains on its network.

The status code is not new — 402 has been reserved in HTTP for this purpose since the beginning and was essentially unused. What is new is an intermediary large enough that a crawler operator has to decide how to respond to it, which is the condition a payment protocol needs to exist at all.

Whether this becomes the norm depends entirely on whether the large crawler operators implement the paying half. That is not something a site owner controls and not something anybody can currently predict, so the practical posture is to keep the option available — which mostly means not signing away access in a way you cannot reverse.

Deciding, with the trade-offs stated

The decision has three inputs and they pull in different directions.

1. What does a click from search or an assistant answer earn you? If it converts to subscriptions, indexing is your acquisition channel and closing it is expensive. If it does not, the calculation is different and honest.
2. What is the marginal harm of being in a training corpus? For an archive whose value is the archive, real. For a site whose value is being current, smaller — a model trained on last year’s prices competes with nothing you sell today.
3. What can you actually enforce? Be honest here rather than aspirational. If the answer is a robots.txt file, your policy is a request. If the answer is verified-identity edge rules or authentication, it is a control. Design the policy you can enforce, and do not build a business assumption on top of one you cannot.