llms.txt: The Proposal, the Adoption, the Honest Status

llms.txt is a real proposal with a precise format, published by Jeremy Howard of Answer.AI in September 2024. What it is not, at the time of writing, is a thing any major assistant operator has documented reading. Those are two separate questions and almost every explainer answers only the first.

What the file is

A markdown file at the root of your domain — /llms.txt — whose purpose is to give a language model a curated, cheap-to-read map of your site. The motivating problem is genuine: a model with a limited context window that lands on a documentation site has to spend that window on navigation chrome, cookie banners and HTML it does not need, and it has no way to know which twenty of your four hundred pages actually matter.

The proposal is deliberately modest in scope. It is not an access control, not a licence, and not a request for anything. It is a hand-written index, in markdown because markdown survives conversion to plain text without loss, at a fixed location so it can be found without being linked.

The format, exactly

The specification at llmstxt.org defines a structure that is strict enough to parse and loose enough to write by hand. In order:

1. An H1 with the name of the site or project. This is the only required element.
2. A blockquote containing a short summary, with enough context to understand what the rest of the file is about.
3. Zero or more markdown sections of any kind — paragraphs, lists — giving more detail, with no headings.
4. Zero or more H2 sections, each containing a markdown list of links. Each list item is a link followed optionally by a colon and a description.
5. An optional final H2 named exactly Optional, whose links may be skipped when a shorter context is needed. This is the one piece of semantics in the format, and it is the reason the file is more than a sitemap.

# Multigrid

> An LLM gateway: one API and one balance across many model
> providers, with routing, failover and per-request telemetry.

Prices and model availability change; the catalogue pages are the
authority for both.

## Docs

- [Quickstart](https://multigrid.ai/docs/quickstart): first request in
  five minutes, with the auth header
- [Routing](https://multigrid.ai/docs/routing): how a request picks a
  provider, and what failover does
- [Errors](https://multigrid.ai/docs/errors): every status code the
  gateway returns and what causes it

## Reference

- [Models](https://multigrid.ai/models): the full catalogue with
  context windows and prices per million tokens

## Optional

- [Changelog](https://multigrid.ai/changelog)
- [About](https://multigrid.ai/about)

Note what the format is not. There is no directive syntax, no allow-or-deny, no expiry, and no field for stating terms. Anything claiming that llms.txt controls training is describing a different file.

The other half: markdown beside HTML

The proposal has a second part that gets far less attention and is arguably the more useful one: serve a markdown version of each page at the same URL with .md appended. A client that wants your quickstart without the navigation fetches /docs/quickstart.md and gets exactly the prose.

A widely used community extension is /llms-full.txt, a single file containing the full text of the linked pages concatenated. It is a convention rather than a separate specification, and it has an obvious failure mode: on a large site it becomes a multi-megabyte file that exceeds any context window it was meant to fit into, and it goes stale silently because nothing regenerates it when a page changes.

Generating it from your source of truth

The only version of this file worth having is one you do not maintain. If your pages already come from a structured source — a content directory, a database, an array of route definitions — then the file is a projection of that source and cannot drift from it.

// scripts/llms-txt.mjs — write public/llms.txt from the same data
// the navigation and the sitemap are built from.
import { writeFileSync } from "node:fs";
import { docs, guides, reference } from "../content/index.js";

const HOST = "https://example.com";

function section(title, items) {
  if (items.length === 0) return "";
  const lines = items.map(
    (i) => "- [" + i.title + "](" + HOST + i.path + "): " + i.summary,
  );
  return "## " + title + "\n\n" + lines.join("\n") + "\n\n";
}

const out =
  "# Example\n\n" +
  "> One paragraph saying what this site is and what the links below\n" +
  "> are for.\n\n" +
  section("Docs", docs) +
  section("Guides", guides) +
  section("Reference", reference) +
  section("Optional", [
    { title: "Changelog", path: "/changelog", summary: "Release notes" },
    { title: "About",     path: "/about",     summary: "Who publishes this" },
  ]);

writeFileSync("public/llms.txt", out);
console.log("wrote", out.split("\n").length, "lines");

Run it in the same build step that produces your sitemap. Two rules make the difference between a useful file and a liability:

• Only list what is live. If your content is released on a schedule, filter to released pages exactly as the sitemap does. A curated map pointing at 404s is worse than no map.
• Write the summaries once, in the page data. The description after each link should be the same string the page uses as its meta description and its index entry. Three copies of a summary is two copies that will be wrong.

Publishing it is not the same as reading it

A great many sites now publish one, and several documentation hosting platforms generate it automatically for every site they host. That is a real fact and it is routinely presented as evidence of adoption. It is evidence of publication. The question that matters is whether anything consumes it, and those are different sides of the transaction.

At the time of writing, no major assistant operator documents fetching /llms.txt as part of its retrieval pipeline or its crawl. There is no equivalent of the crawler documentation pages that list agent tokens and address ranges, no mention in the published guidance for site owners, and no status code table. That is not proof that nothing reads it — it is the absence of the documentation that would make the claim tier 1 in the sense used in what is real and what is sold.

The tools that demonstrably do read it are a different category: developer tooling, documentation search, agent frameworks that were pointed at your site deliberately, and one-off scripts. That is a genuine use and it is worth having. It is not web-scale assistant retrieval.

How to find out whether anything fetches yours

You do not have to take anybody’s word for this, including this page’s. The file has a fixed path, so a single log query answers the adoption question for your own domain, and it is the highest-value thing in this article.

# Combined-format access log
grep -E ' /(llms\.txt|llms-full\.txt)[ ?]' /var/log/nginx/access.log \
  | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

# Caddy or any JSON-lines log
jq -r 'select(.request.uri | test("^/llms(-full)?\\.txt"))
       | .request.headers["User-Agent"][0]' access.log \
  | sort | uniq -c | sort -rn

Run it over a month of logs. You will get one of three answers: nothing at all, a handful of requests from tools and curious humans, or sustained requests from an identifiable agent. The third would be genuinely newsworthy, and it would be verifiable by anyone else with the same query — which is what a tier-2 fact looks like. Do the same for the request rate against your ordinary pages to see the ratio.

Keep in mind that a fetch is not a use. Something can request the file and discard it. The log query establishes the ceiling of possible adoption, not the floor of actual effect.

What it cannot do

• It cannot stop anything. There is no deny syntax. Access control for crawlers is robots.txt, a firewall rule, or an authentication check — and only the last two are enforcement.
• It cannot state terms. It carries no licence field and no price. Arrangements about use are contracts, not files; see how licensing deals are structured.
• It cannot make a page rank. It is not read by the search backends that supply candidates to assistants, which is the stage that decides whether you are in the running at all.
• It cannot fix an unreadable site. If your pages need JavaScript to show their text, a curated list of those pages is a curated list of blanks.

Whether to publish one anyway

Yes, on one condition: that it costs you almost nothing and is generated rather than maintained. A file that is derived from the same data your navigation is derived from stays correct for free. A hand-written one becomes a page of dead links within two release cycles, and a dead link in a file whose entire purpose is to be trusted as a map is worse than no file.

Treat it the way you would treat a well-written README: cheap, useful to a human who wants the shape of your site in thirty seconds, and possibly useful to a machine later. Do not treat it as a ranking mechanism, and do not pay anybody to write one. The claim that it changes assistant behaviour is tier 3 and currently has nothing behind it.