llms.txt, explained for humans

llms.txt is a Markdown file you put at the root of your site, at /llms.txt, describing what your site is and which pages are worth reading. It is the newest member of a small family that already includes robots.txt and sitemap.xml, and it exists because those two answer the wrong questions for a language model.

robots.txt says what a crawler may fetch. sitemap.xml says what exists. Neither says what any of it means, or which twelve of your four hundred URLs actually explain the product.

What goes in it

Keep it short. The format is deliberately plain:

# Crawld

> An SEO and GEO scorecard that ships verified fixes behind your approval.

## Docs
- [How we measure](https://crawld.co/how-we-measure): the scoring method and
  why unmeasured is never counted as a pass
- [The crawler](https://crawld.co/bot): what our bot fetches and how to block it

## Product
- [Pricing](https://crawld.co/pricing): plans and what each includes

A heading with your site name, a blockquote summarising it in one sentence, then grouped links with a short gloss on each. That is the whole specification worth caring about.

What does not go in it

• Every URL you have. That is what a sitemap is for. This file is a curation, and its value comes entirely from what you left out.
• Marketing copy. A model reading this is trying to work out what you do. “Supercharge your workflow” tells it nothing.
• Anything untrue. It is a public file. Claims in it are claims.

Does it actually work?

Honestly: partially, and unevenly. Adoption is real but not universal, and no major engine has committed to honouring it the way they honour robots.txt. You should treat it as cheap insurance rather than a lever.

What makes it worth the ten minutes is that the work is not wasted if adoption stalls. Writing an llms.txt forces you to answer “which pages on this site actually explain what we do?”, and most teams discover the answer is fewer pages than they thought, and that two of them are out of date.

Why your site builder didn’t make one

Because it is new, because it is not required, and because nothing breaks without it. The same reasons your meta description is empty.

Generating one is straightforward if you already know your page inventory and which pages matter, which is the same inventory a site audit produces. That is the connection: this file is a by-product of knowing your own site structure, and most sites do not.

Scan your site free All posts