llms.txt: what the spec actually says

Someone showed me their llms.txt last month and it was 380 KB. They'd concatenated every page of their documentation into it, feeling appropriately thorough, and asked me why nothing seemed to be reading it.

I went and reread the spec, which I'd assumed I remembered. I didn't. Here's what it actually says.

llms.txt is a Markdown file at your site root that gives AI systems a curated map of your documentation. Per the specification, proposed by Jeremy Howard on September 3, 2024, exactly one element is required: an H1 with the site's name. Everything else, including the summary and the link lists, is optional. The whole thing can legitimately be four lines long.

What the llms.txt spec actually requires

The structure, in order:

1. An H1 with your project or site name. This is the only required element.
2. A blockquote summary with the key information about the project.
3. Zero or more Markdown sections of free-form detail.
4. Zero or more H2-delimited file lists. Each list item is a required Markdown link, then optionally a colon and notes: - [name](url): notes.

That's it. A valid, useful file looks like this:

# Acme API

> REST API for scheduling and dispatch. Authentication is API-key based.

## Docs

- [Quickstart](https://docs.acme.com/quickstart): Get a key and make your first call
- [Authentication](https://docs.acme.com/auth): Key scopes, rotation, and errors

## Optional

- [Changelog](https://docs.acme.com/changelog): Release history

One detail worth knowing: the H2 named Optional has defined meaning in the spec. URLs listed under it can be skipped when a shorter context is needed. It's the one place you get to tell a machine what to drop first when it runs out of room, and it's free to use. Most files I look at don't have it.

The part everyone skips

Here's the recommendation that gets ignored, and it's the one that does real work: the spec says to provide a Markdown version of each page at the same URL with .md appended. So /quickstart also serves /quickstart.md. URLs without a filename append index.html.md instead.

Think about what happens without it. An agent reads your beautifully curated llms.txt, follows a link, and gets HTML. Now it's parsing your nav, your cookie banner, your sidebar, your footer, and your marketing CTA to find three paragraphs of actual content. Everything you carefully organized in the index gets undone at the moment of retrieval.

The .md mirror is what makes the link worth following. llms.txt is a table of contents, and a table of contents is only as useful as the pages it points at.

llms-full.txt is not in the spec

This surprised me, so I'll state it plainly: llms-full.txt does not appear in the llms.txt specification. It's a community convention that grew up alongside it. The spec mentions llms-ctx.txt and llms-ctx-full.txt, but as processing outputs generated from an llms.txt file, not as files you're expected to publish.

That doesn't make llms-full.txt useless. It has a genuinely different job: one file containing your entire documentation, sized to be pasted or piped into a context window in one shot. That's a real workflow and worth serving.

It's just not a substitute for llms.txt, and it's the wrong thing to make enormous. A 380 KB concatenation of everything is not a map. It's the territory, flattened.

Is it actually worth doing?

The honest answer: nobody can promise you that publishing an llms.txt gets you cited more often. It's a proposal, not a ratified standard, and the major AI companies have not collectively committed to consuming it. Anyone telling you it's a guaranteed ranking factor for AI answers is selling something.

I'd still ship it, for two unglamorous reasons.

It costs almost nothing. A hand-written llms.txt for a small docs site is twenty minutes, and generated ones are free.

And the .md mirrors pay off regardless of whether a single crawler ever reads your index. Anything retrieving your docs, an agent, a RAG pipeline, someone's internal script, gets clean Markdown instead of scraped HTML. That benefit doesn't depend on the spec winning.

Then measure it rather than assuming. Your server logs will tell you which AI crawlers actually request these files, and that's worth watching on your own docs instead of taking my word for it.

How this works in Doccupine

Everything above is generated for you, which is why I've spent time in this spec.

npx doccupine writes llms.txt in spec form: H1 site name, blockquote description, H2 sections with - [Title](url): description entries organized by your section and category structure. It writes llms-full.txt alongside it. And it writes a .md mirror for every page, each one carrying the page title, description, a Source: URL, and the body as clean Markdown.

All three regenerate whenever your content changes, and stale .md files from deleted pages get pruned rather than left behind to serve outdated content to agents. If you want the deeper version of why any of this matters, I wrote about what makes documentation readable by AI agents separately.

The person with the 380 KB file wasn't wrong to want to be thorough. They just built the bulk download and called it the index. Those are different files with different readers, and the four-line version was going to work better than the one that took them an afternoon.

If you've got server logs showing which AI crawlers request llms.txt versus .md pages versus plain HTML, I'd genuinely like to see the ratio. Email [email protected]. That data is the only real answer to "is this worth doing," and almost nobody publishes it.