The robots.txt Decision Framework for AI Crawlers

robots.txt, briefly: a real standard, not a suggestion

robots.txt is a plain text file at the root of a website that tells visiting software which parts of the site it may and may not crawl. It isn't a proprietary Google invention — it's formally standardized as the Robots Exclusion Protocol, RFC 9309, published through the Internet Engineering Task Force, the body that standardizes how the internet's basic plumbing works.

Every major AI crawler operator states it respects this file for its automated crawling bots — OpenAI for GPTBot and OAI-SearchBot, Anthropic for ClaudeBot and Claude-SearchBot, and Perplexity for PerplexityBot all say so in their own documentation. The user-triggered bots — ChatGPT-User, Claude-User, Perplexity-User — are the exception, since they act on a specific person's direct request.

Question one: do you want AI assistants recommending you?

This sounds obvious, but it's worth stating plainly because the honest answer differs by business. A dentist, a plumber, a local restaurant — almost anyone whose customers ask friends, and increasingly AI assistants, "who should I use" — generally wants the answer to be yes. A business built on exclusivity or a tightly controlled sales process might have real reasons to say no.

If the answer is yes, that points toward allowing the crawlers that power an assistant's answers: the training crawlers that teach a model about your existence, and the search-style crawlers that let it cite you directly.

Question two: does this content make you money directly?

Some content exists purely to be found and read — a services page, a blog post, an FAQ. Other content is the product itself: a news publisher's subscriber-only articles, a SaaS company's premium documentation, a marketplace's detailed listings. Content in the second category deserves a more careful answer, because letting it train someone else's model gives away the thing you're trying to sell, rather than advertising it.

This question often splits a single website into two different answers: the marketing pages say yes to crawling, and the paid or proprietary content says no — which is entirely legitimate, since robots.txt rules can be written for specific paths, not just the whole site.

Question three: can you actually maintain this?

A robots.txt file that names ten specific bots today will be out of date the day an eleventh one launches. If nobody on your team is going to revisit this file every few months, a simpler, more durable policy — allow the well-documented major crawlers as a category, revisit twice a year — beats an elaborate rule set nobody maintains.

Being realistic about your own maintenance capacity isn't a compromise; a policy you can actually keep current protects you better than an ambitious one that goes stale within a year.

Working the framework through three real businesses

A local plumber answers yes to question one (wants AI recommendations), the question barely applies for question two (a services page earns nothing by being kept secret), and question three is easy (there's little to maintain). The framework lands cleanly on: allow the major crawlers broadly.

A SaaS company answers yes to question one for its marketing site and public blog, but a careful no to question two for its premium documentation and in-product help content, which it sells access to. Its robots.txt ends up with two different rules for two different parts of the same domain — open marketing pages, closed paid docs.

A news publisher often gives the most layered answer: yes to question one for its free articles, since citations drive real reader awareness, but a firm no to question two for paywalled investigative work its business depends on. Because Google-Extended and its equivalents at other companies separate training from search visibility, a publisher can plausibly allow search-style crawling of free content while still declining to have that same content used for model training.

FAQ