JavaScript Rendering and What Crawlers See

A crawler that does not execute JavaScript sees your HTML response and nothing else. If your text arrives via a client-side fetch, that crawler sees an empty shell, gets a 200, and reports no error anywhere. This is measurable on your own site in about two minutes.

The gap, and why it is the most expensive bug here

Three versions of your page exist and they can all differ: the HTML your server returns, the DOM after JavaScript has run, and the text a converter extracts from one of those. Every discussion of crawling and retrieval is really about which of the three a given client got.

The reason this is the most expensive failure in the cluster is that it is completely silent. There is no error, no warning, and no report. Your page works perfectly in a browser, your monitoring is green, your status code is 200, and a whole class of clients has been receiving a navigation bar and a loading spinner for a year.

Who renders and who does not

The major search crawlers render. Googlebot uses a recent Chromium and runs your JavaScript, though it does so in a second pass that can lag the initial crawl, which means content that depends on rendering is indexed later than content that does not — sometimes much later. Bingbot also renders.

The evidence on AI fetchers is thinner and is one of the few places where a published third-party measurement exists: Vercel published an analysis in December 2024 of AI crawler behaviour observed across its network, reporting that the AI-assistant fetchers it observed did not execute JavaScript, in contrast to the search crawlers. That is a named, dated study of one network rather than a universal law, and it is the kind of thing that can change without announcement.

Which is exactly why the rest of this page is a measurement rather than a table. You do not need to know whether a given fetcher renders if your content does not depend on rendering. Close the gap and the question stops mattering.

The second pass, and the lag it costs

“Googlebot renders JavaScript” is true and is not the end of the sentence. Crawling and rendering are separate stages: the crawler fetches your HTML, and the page then joins a render queue to be processed by a headless browser at some later point. Indexing happens after that second pass for anything that only exists after rendering.

The queue delay is not published and varies with how much a site is worth rendering, so the honest statement is that it is measured in hours to days rather than in milliseconds. The consequence is consistent even without a number: server-rendered content is available at crawl time and client-rendered content is available at render time, and those are different days. For a page about something time-sensitive — a price, an availability, a release — the gap is the whole value.

Framework specifics, because this is where the decision is made in practice. In a framework with a server-component boundary, everything above that boundary is in the HTML response and everything below it is not. A component marked as client-side that fetches data on mount produces exactly the gap this page is about. The rule that keeps you out of trouble is to draw the boundary around interactivity rather than around convenience: a filter control belongs on the client, and the list it filters usually does not.

Streaming complicates the picture in a way worth knowing. A page that streams sections as they resolve does eventually put all of them in the response body, so a client that reads the response to completion gets everything — but a fetcher that gives up early, or one that treats the first flush as the document, may not. If you stream, run the measurement below against the streamed page specifically rather than assuming it behaves like the static one.

Measuring your own gap

The quickest version, in two commands. Fetch without rendering, count the words; then compare against what you see in a browser.

# What a non-rendering fetcher gets
curl -sS -A "OAI-SearchBot" https://example.com/some-page \
  | sed -e 's/<script[^>]*>.*<\/script>//g' \
        -e 's/<style[^>]*>.*<\/style>//g' \
        -e 's/<[^>]*>/ /g' \
  | tr -s ' \n' ' ' | wc -w

# Sanity check: does a phrase from the middle of your article exist
# in the raw response at all?
curl -sS https://example.com/some-page | grep -c "a phrase from paragraph six"

If that grep returns 0 and the phrase is plainly on the page in your browser, you have found the problem and you can stop reading the measurement section.

The diff script

The proper version renders with a headless browser and reports the delta. Requires Playwright (npm i -D playwright and npx playwright install chromium).

#!/usr/bin/env node
// render-gap.mjs — how much of the page needs JavaScript?
//   node render-gap.mjs https://example.com/page [more urls...]

import { chromium } from "playwright";

const UA_BOT = "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; " +
               "+https://openai.com/searchbot)";

function textFromHtml(html) {
  return html
    .replace(/<script[\s\S]*?<\/script>/gi, " ")
    .replace(/<style[\s\S]*?<\/style>/gi, " ")
    .replace(/<noscript[\s\S]*?<\/noscript>/gi, " ")
    .replace(/<[^>]+>/g, " ")
    .replace(/&nbsp;/g, " ")
    .replace(/\s+/g, " ")
    .trim();
}

const words = (s) => (s ? s.split(" ").filter(Boolean).length : 0);

const urls = process.argv.slice(2);
if (urls.length === 0) { console.error("usage: render-gap.mjs <url>..."); process.exit(1); }

const browser = await chromium.launch();

console.log("raw   rendered  kept%  jsonld  url");
for (const url of urls) {
  // 1. The unrendered response, requested as a bot.
  const res = await fetch(url, { headers: { "User-Agent": UA_BOT } });
  const html = await res.text();
  const rawText = textFromHtml(html);
  const rawJsonLd = (html.match(/application\/ld\+json/g) ?? []).length;

  // 2. The rendered DOM.
  const page = await browser.newPage({ userAgent: UA_BOT });
  await page.goto(url, { waitUntil: "networkidle" });
  const domText = (await page.innerText("body")).replace(/\s+/g, " ").trim();
  const domJsonLd = await page.locator(
    'script[type="application/ld+json"]').count();
  await page.close();

  const r = words(rawText), d = words(domText);
  const kept = d === 0 ? 0 : Math.round((r / d) * 100);
  console.log(
    String(r).padStart(5),
    String(d).padStart(9),
    String(kept).padStart(6) + "%",
    (rawJsonLd + "/" + domJsonLd).padStart(7),
    " " + url,
  );

  // 3. What is in the DOM and missing from the raw response? Show a sample.
  if (kept < 90) {
    const rawSet = new Set(rawText.toLowerCase().split(" "));
    const missing = domText.split(" ")
      .filter((w) => w.length > 6 && !rawSet.has(w.toLowerCase()));
    console.log("      missing sample:", [...new Set(missing)].slice(0, 12).join(" "));
  }
}

await browser.close();

The jsonld column is there deliberately: structured data injected client-side is invisible to a non-rendering fetcher for exactly the same reason the prose is, and it is a common oversight because the validator you tested with rendered the page.

Reading the number

Also check the raw-vs-rendered JSON-LD count. A ratio of 0 to 1 means your structured data does not exist for anything that does not render, which quietly undoes the work in structured data that machines read.

Closing the gap

1. Render the content on the server. Server-side rendering or static generation for anything that is content rather than interface. In a framework with server components, the rule of thumb is that anything rendering text a stranger should read belongs on the server side of the boundary.
2. Do not fetch your own content from your own client. A component that mounts and then fetches the article body is the single most common cause of this problem, and moving the fetch to the server is usually a small change.
3. Put structured data in the server response as a plain script element in the initial HTML, not injected after hydration.
4. Make tabs and accordions present in the DOM. Hidden with CSS is fine and extracts fine; not-yet-fetched is not.
5. Use real links. A div with a click handler is not navigable by anything that does not run your JavaScript. An a with an href is, and it also works for keyboards and screen readers.
6. Paginate with URLs. Infinite scroll with no underlying paged URLs means everything past the first screen is unreachable to a fetcher.
7. Put the render check in CI against a handful of representative URLs, failing the build below a threshold. This class of bug returns with every refactor.

Five traps that survive a rewrite

• Consent walls. A cookie banner that blocks content until dismissed can leave a fetcher with a banner and nothing else. Check what an unconsented, non-interacting client actually receives.
• Bot detection that fires on unusual user agents. Your WAF may be serving a challenge page to the exact clients you care about, with a 200 and no content. The script above will show it as a very low word count with plausible-looking navigation.
• Geographic redirects. A crawler fetching from an unexpected region gets bounced to a country selector.
• Soft 404s. A missing page returning 200 with “not found” in the body. Nothing downstream can tell that from content.
• Lazy-loaded text. Images can be lazy-loaded safely; text below the fold that only loads on scroll cannot, because no fetcher scrolls.

All five produce a page that is perfect in your browser and empty to a crawler, and all five are caught by running the measurement rather than by reasoning about the code. That is why this page ends in a script and why the same check appears as an item in the machine-readability audit.