Programmatic Pages Without Becoming a Content Farm

Generating pages from data is not the problem; Google’s own spam policy names scaled content abuse, and the word doing the work is abuse. The line is testable: strip the variables out of two of your generated pages and see whether anything of value remains on either side of the substitution.

What the policy actually says

In March 2024 Google introduced a spam policy named scaled content abuse, replacing an earlier and narrower automatically-generated-content rule. The change that matters is that the new wording is about purpose and value rather than about method: pages generated at scale primarily to manipulate rankings and without providing value to users. Generation by a machine is not itself the violation, which is the part most summaries get backwards.

That leaves you with a judgement rather than a rule, and judgement is uncomfortable to build a system on. So the rest of this page converts it into something you can run.

The test: strip the variables and look

Take two pages from your generator that differ as much as any two of them do. Delete every value that came from your data — the names, the numbers, the places. Now read what is left.

• If the remainder is prose that is identical on both, and the deleted values were the only difference, you have a template whose pages are interchangeable. That is the failure case regardless of how many pages it produced.
• If deleting the values destroyed the page — because the values were the substance, and a reader came specifically to get them — you have a reference page. That is the success case, and it is what a catalogue, a specification table or a price list is.

The distinction is not about word count and it is not about whether a model wrote the prose. It is about whether the generated page contains a fact that is true only of that page and that the reader would otherwise have had to find or compute themselves.

A template with something in it

One page per model in a catalogue. Each carries the context window, the price per million input and output tokens, which providers serve it, the modalities it supports, and whether it supports structured output and tool calls. All of it read from a database at request time, so the page is correct rather than correct-as-of-build.

/models/<vendor>/<model>

  Context window       200,000 tokens
  Input                $3.00 per million tokens
  Output               $15.00 per million tokens
  Served by            4 providers
  Modalities           text, image in
  Structured output    yes
  Tool calling         yes

  Plus: which provider is cheapest right now, what the
  price was last month, and the same table for the three
  nearest alternatives.

Strip the variables and almost nothing remains — which is the point. Every page answers a question somebody actually typed (“<model> pricing”, “<model> context window”) with a current, specific answer that is tedious to assemble by hand. Three hundred such pages is three hundred distinct answers, not three hundred copies.

Two properties make it work beyond the data itself. The data is current, so the page is worth revisiting rather than being a snapshot. And the page does something with the data — a comparison, a cheapest-now, a derived figure — so the reader gets more than a row of the table they could have downloaded.

A template with nothing in it

/services/<service>-in-<city>

  "Looking for professional <service> in <city>? Our expert
   <service> specialists in <city> provide reliable <service>
   services throughout <city> and the surrounding area. Contact
   our <city> <service> team today."

  8 services × 400 towns = 3,200 pages

Strip the variables and one sentence remains, repeated 3,200 times. The reader learns nothing that depends on the city or the service; the page exists to match a query string rather than to answer it. That is scaled content abuse in the policy’s own terms, and the volume is what makes it visible.

The interesting version of this failure is the near miss: a template that could be a reference page but whose data is thin. City-level pages with the actual local response time, the actual practitioners, the actual price for that market and the actual regulations that apply there are reference pages. The same template with only the city name substituted is not. The difference is entirely in whether you have the data, and no amount of prose generation substitutes for not having it.

Measuring your own similarity

The test above can be automated. Compare generated pages pairwise using shingled Jaccard similarity — the proportion of overlapping word sequences — and look at the distribution.

#!/usr/bin/env node
// template-similarity.mjs — how much of a generated page is boilerplate?
//   node template-similarity.mjs page1.txt page2.txt page3.txt ...
//
// Feed it EXTRACTED TEXT, not HTML. Shingle size 5 is a reasonable
// default: long enough to ignore common phrases, short enough to catch
// a reworded template.

import { readFileSync } from "node:fs";

const N = 5;

function shingles(text) {
  const words = text.toLowerCase().replace(/\s+/g, " ").trim().split(" ");
  const set = new Set();
  for (let i = 0; i + N <= words.length; i++) set.add(words.slice(i, i + N).join(" "));
  return set;
}

function jaccard(a, b) {
  let shared = 0;
  for (const s of a) if (b.has(s)) shared++;
  return shared / (a.size + b.size - shared);
}

const files = process.argv.slice(2);
const docs = files.map((f) => ({ f, s: shingles(readFileSync(f, "utf8")) }));

const scores = [];
for (let i = 0; i < docs.length; i++)
  for (let j = i + 1; j < docs.length; j++)
    scores.push([jaccard(docs[i].s, docs[j].s), docs[i].f, docs[j].f]);

scores.sort((a, b) => b[0] - a[0]);
const median = scores[Math.floor(scores.length / 2)][0];

console.log("pairs compared:", scores.length);
console.log("median similarity:", median.toFixed(3));
console.log("\nmost similar pairs:");
for (const [s, a, b] of scores.slice(0, 10))
  console.log("  " + s.toFixed(3), a, b);

The thresholds are a rule of thumb rather than a published standard, and they are stated as such. Their value is comparative: run the script before and after cutting your boilerplate and watch the median move. That is a measurement of your own change, which is the kind of number this cluster trusts.

Publication rate is its own signal

A separate risk that has nothing to do with page quality: publishing a thousand URLs in a week is itself a pattern associated with scaled abuse, and a site that does it can lose trust it had earned page by page. Ten to twenty pages a week is a comfortable ceiling for a site building a library.

The structural fix is to separate writing from publishing. Give each page a release date; make the index, the hub pages and the sitemap read only the released ones; and have an unreleased URL return a 404 rather than render. Then a hundred finished pages can sit in the repository, be reviewed, be committed and be deployed without any of them existing to a visitor or a crawler until its date arrives.

The important detail is which way the default points. A page with no date should be treated as unpublished, not as published immediately. Both failure modes are real — forget a date and one page is invisible, or forget a date and the entire backlog goes live on merge — and only the first is recoverable. Make the second one impossible and have your test suite fail on a page in that state, so it is loud rather than quiet.

Seven rules that keep a generator honest

1. Generate only where you have data. If the data for a page is thin, the page should not exist. A generator that emits a page per row should be allowed to emit nothing.
2. Put the data first. The specific values should be near the top, in the first passage a chunker would take.
3. Read the data at request time where it changes. Prices baked in at build time are prices that can be wrong at read time, and a wrong price is worse than no page.
4. Derive something. A comparison, a ranking, a per-unit figure. Derivation is what makes the page more than a row of a table.
5. Cut the boilerplate until it hurts. Every shared sentence appears on every page. If it is not necessary on all of them, it is not necessary on any.
6. Set a floor and enforce it in CI. A minimum count of distinct data points per page, and a maximum pairwise similarity. Fail the build rather than reviewing three thousand pages by hand.
7. Release on a schedule. See above; merging should not be publishing.