Retrieval-Friendly Site Architecture

A retrieval system does not evaluate your site, or even your page. It cuts everything into passages of a few hundred tokens and evaluates those. Almost every architectural decision worth making follows from taking that literally.

Your site is going to be cut into pieces

Whatever fetches your page converts it to text and splits it. The splitters in common use fall into three families, in increasing order of respect for your intentions: fixed character or token windows with overlap; recursive splitting that prefers paragraph and sentence boundaries; and structure-aware splitting that cuts at headings and carries the heading trail into each piece.

You do not know which one is being used on you, and it differs between systems and over time. So the architecture that works is the one that survives all three: sections short enough that a fixed window rarely has to cut mid-argument, paragraphs that read alone if it does, and headings descriptive enough to be useful when they are the only context a chunk carries. The detail on chunking itself is in chunking strategies.

One canonical answer per URL

This is the single highest-leverage structural rule and it is violated constantly, usually for reasons that sounded good.

If three of your pages each half-answer “how do I rotate an API key”, then for that question you have no strong candidate — you have three weak ones competing with each other for the same retrieval slot, and the passage that wins is a partial answer. Consolidating them into one page that answers it completely produces one strong candidate. The classical search version of this problem has been understood for years; the retrieval version is more severe, because a partial answer is not just a worse result, it is a wrong one when quoted.

The practical policy:

• Every question your users ask maps to exactly one URL. Write the question list down; it is the site map that matters.
• Consolidate rather than cross-link when two pages overlap substantially. Redirect the weaker one; do not leave both live with a “see also”.
• Canonicalise aggressively. Trailing slashes, case, query-string variants, and print or AMP-style alternates all create near-duplicate candidates that split your evidence.
• Do not paginate an answer. An answer split across three pages is an answer no single passage contains.

How long a page should be, derived

There is no universal number, but there is arithmetic that gives you a target for your own content.

Assume a mid-sized chunk:        600 tokens
English prose:                   ~4 chars per token
                                 --------------------
One chunk:                       ~2,400 characters
                                 ~380 words

A section that answers one question in <= 380 words
almost always arrives whole.

A 1,500-word page split into 4 sections of ~375 words
gives 4 self-contained candidates from one URL.

The same 1,500 words as one undivided block gives
~4 chunks whose boundaries fall wherever the splitter
happens to land — typically mid-argument.

So the target is not a page length; it is a section length. Aim for sections of roughly 200 to 400 words, each answering one thing completely. The page can then be as long as the subject deserves, because length is being carried by the number of self-contained sections rather than by the size of any one of them.

This also resolves the perennial argument about long pages against short ones. A long page of well-bounded sections is many candidates under one URL, which is strictly better than the same sections scattered across many thin URLs — those compete with each other, dilute whatever authority the URL has accumulated, and each one carries its own navigation overhead into the extracted text.

Headings are the chunk boundaries

Structure-aware splitters cut at headings and often prepend the heading trail to each chunk, which makes your heading text part of the passage a scorer sees. Four rules follow directly:

1. One h1 per page, and it should be the question the page answers.
2. Never skip a level. An h2 followed by an h4 produces a nesting the splitter reconstructs incorrectly.
3. Make every h2 a claim or a question, not a label. “Configuration” adds nothing to a chunk; “Which timeout applies when both are set” adds the subject.
4. Do not use headings for visual weight. A bolded paragraph styled as a heading is not a boundary, and a heading used for emphasis creates one where you did not want it.

URLs that survive

A citation is a URL. That gives URL stability a value it does not have in ordinary site design: a URL that changes loses not only its links but its accumulated presence in whatever corpora and indexes have recorded it, and unlike a link, a training corpus cannot follow your redirect.

• Descriptive and short. The path is visible in a citation and readers judge it.
• No dates in the path unless the content is genuinely an event. A dated URL for evergreen content forces a choice between a stale URL and a migration.
• No session identifiers, no tracking parameters in internal links. They create duplicate candidates.
• Pick one form and enforce it. Trailing slash or not, lowercase, one host. Redirect the rest in one hop, permanently.
• When you must move a page, keep the redirect forever. Not for a year. The old URL will be in corpora for longer than your redirect map is likely to survive a migration, which is an argument for not moving pages.

Hubs, spokes and the links between them

A connected body of work on one subject is easier to retrieve from than the same pages as orphans, for a mechanical reason as well as a reputational one: internal links are how a crawler discovers depth, and a hub page is the single URL most likely to be fetched when the topic rather than the question is the query.

What makes a hub useful rather than decorative: it lists every spoke with a sentence saying what that spoke answers, so the hub itself is a retrievable passage that maps questions to URLs. A hub that is a grid of titles is a navigation element; a hub whose entries are described is a document.

For a site publishing continuously, there is a second consideration that has nothing to do with retrieval and everything to do with how bulk publication is read: shipping several hundred pages in a week is a pattern search operators treat as a scaled-content signal. Releasing on a schedule rather than on merge is the structural fix, and it is discussed with the rest of the template question in programmatic pages without becoming a content farm.

Dates, and what a crawler does with them

Three date signals exist and they should agree: a visible date on the page, dateModified in your structured data, and lastmod in your sitemap. Two rules about them.

Only publish a date you actually maintain. A lastmod generated at render time claims every page changed on every fetch, which teaches a crawler to ignore the field entirely. Search operators have said they use lastmod where a site is consistently accurate with it, and that consistency is the whole precondition.

Do not date pages you did not change. Bumping a date to look fresh is the kind of small dishonesty that costs you the value of the signal at exactly the moment you need it — when you have genuinely revised something and want it recrawled.

Where a page contains a figure that will move, say when it was true in the sentence containing it, rather than relying on a page-level date that a chunk will not carry. That is the same principle as attaching a number to its conditions, applied at the level of the site.