llms.txt: What It Is and How to Write One

What the proposal actually says

The problem llms.txt was designed for is context, not crawling. When a model or an agent needs to answer a question using your site, it has a limited context window and a hostile input: HTML pages full of navigation, cookie banners, sidebars, ads and JavaScript-rendered content. Converting a large site into something a model can reason about is lossy and expensive.

The llms.txt specification proposes two complementary fixes. First, a single Markdown file at /llms.txt that acts as a curated index — here is what this project is, here are the pages that matter, in priority order. Second, publishing clean Markdown versions of individual pages at predictable URLs (a page at /docs/setup also available at /docs/setup.md), so that once an agent has picked a link from your index it can fetch text rather than a template.

Note what it is not. It contains no directives, no allow or disallow rules, and no permissions. It is a reading list you offer, and any consumer is free to ignore it. The spec is deliberately Markdown rather than JSON or XML precisely because the intended consumer is a language model, and because humans need to be able to maintain it by hand.

• Where adoption actually stands
• paragraphs
• bullets

Where adoption actually stands

Be sceptical of anyone claiming llms.txt is now a ranking factor. Google's John Mueller has publicly compared it to the old keywords meta tag and stated that Google does not use it for Search, coverage of which you can find at Search Engine Roundtable. Google's own crawling and indexing documentation makes no mention of the file; the file it does document for crawler control is robots.txt.

On the consumer side, the honest position is that no major assistant vendor has published documentation saying its crawler or its retrieval layer uses /llms.txt to decide what to fetch or cite. Where the file is demonstrably used is the other direction: developers paste an llms.txt URL into a chat or point a coding agent at it, and the agent follows the links. That is a real workflow, and it is the workflow the spec was written for.

Publisher-side adoption is easiest to see in developer documentation, partly because documentation platforms such as Mintlify generate the file automatically for the sites they host. That is a good reason to have one if you ship docs — and a bad reason to assume the ecosystem has standardised. A convention that is trivially cheap to adopt tells you little about whether anything is consuming it.

How to write llms.txt: the exact format

The spec defines a precise document order, and the ordering is the whole point — a consumer should be able to parse it with a simple Markdown reader rather than heuristics.

The one required element is an H1 containing the name of the site or project. Everything after that is optional but positionally constrained: an optional blockquote immediately after the H1 holding a short summary with the key information needed to understand the rest; then zero or more Markdown sections of prose or lists with additional detail, containing no headings; then zero or more H2 sections, each holding a list of links.

Link lines take the form of a Markdown list item with a hyperlink and an optional description after a colon: the name, the URL, then a sentence explaining when to read it. Those descriptions do real work — they are how a model decides which two of your fifteen links to spend context on, so write them as routing instructions, not marketing copy.

One H2 has special meaning: a section titled Optional. Links placed under it may be skipped when a shorter context is needed. Use it for changelogs, deep reference material and anything a reader could reasonably answer without. A separate convention, llms-full.txt, holding the full concatenated text of your documentation, is popular with docs platforms but is not part of the core spec — treat it as an extra, and only publish it if your content genuinely fits in a modern context window.

• Serve it at the domain root as /llms.txt, as text/plain or text/markdown, over HTTPS.
• Keep it short. Ten to thirty curated links beats a sitemap dump; if you needed every URL, you already have an XML sitemap.
• Write descriptions that disambiguate: “pricing tiers and limits”, not “learn more about Growli”.
• Only link pages that are publicly accessible and stable. Broken links in a curated index are worse than no index.
• Regenerate it in your build, not by hand, if your docs change weekly.

A worked example: Growli's own file

Ours is deliberately small. Growli measures how AI assistants mention and recommend businesses, so the pages an agent needs are: what the product does, how the measurement works, pricing, and the research posts that define our terms. Nothing else earns a slot.

Read as a file, it is an H1, a one-blockquote summary, then two H2 sections — the second being Optional. Each line below is a line of that file:

• # Growli
• > Growli tracks how AI assistants — ChatGPT, Gemini, Claude, Perplexity, Copilot and Grok — mention, compare and recommend businesses, and turns the gaps into prioritized actions.
• ## Core
• How it works: the measurement method — prompt sets, run cadence, and how mentions and recommendations are scored.
• AI visibility: definition of the metric, what moves it, and what does not.
• ## Optional
• Blog: research posts on assistant behaviour, citation patterns and measurement methodology.

Where it fits next to robots.txt and structured data

These three files answer three different questions, and conflating them is the most common mistake we see in audits.

robots.txt answers may you fetch this — it is a crawler access-control file with a long-standing, documented parsing behaviour, and it is where you allow or disallow named AI crawlers. If you want to restrict training or crawling, that happens there and in your terms, never in llms.txt. Structured data answers what is this — Schema.org vocabulary in JSON-LD lets you state unambiguously that you are an Organization with this name, these products, this price and these reviews, which is the kind of machine-readable fact that survives being summarised. llms.txt answers only what should I read first, for a consumer that has already decided to read you.

In terms of return on effort, the ordering is unglamorous but consistent: fix crawlability and server-rendered content first, then entity-level structured data, then llms.txt. A file listing links to pages that render blank without JavaScript helps nobody.

So should you publish one?

Yes, if you have documentation, an API, a developer product or a knowledge base — the cost is minutes, the file is human-legible, and the workflow where someone hands an agent your llms.txt URL is already common. Publish it, keep it accurate, and move on.

Probably not worth prioritising if you are a local service business, an e-commerce catalogue or a marketing site with twelve pages. There is no evidence it changes whether an assistant recommends you, and the effort is better spent on the things assistants demonstrably do read: pages that render without JavaScript, explicit comparison and pricing content, consistent entity facts across your site and third-party listings, and coverage in the review sites and roundups models lean on when they answer “best X for Y”.

The honest test is measurement. Track how often assistants name you, on which prompts, and which sources they cite when they do — that is what our AI visibility tracking is built for, and how it works explains the method. If publishing llms.txt moves those numbers for your site, you will see it. If it does not, you will have spent twenty minutes and learned something true.

FAQ