Should You Block AI Crawlers? The Real Tradeoffs for Your Business

What blocking actually protects

The clearest, most defensible reason to block an AI crawler is training-data control. If a crawler never sees your content, that specific content can't become part of what a model learned from — this is a direct, mechanical fact, not a matter of degree. For a business whose content is its product — a paid course, a research report, journalism behind a paywall — this is a real and legitimate protection, not an overreaction.

Blocking a crawler also removes whatever small server load that crawler's requests add, though for most small and mid-sized business websites, this load is unlikely to be noticeable next to ordinary human traffic, and shouldn't be the deciding factor on its own.

What blocking actually costs

The cost is the flip side of the training-data protection: a business people increasingly discover through an AI assistant's answer, rather than a traditional search result, simply cannot be mentioned by an assistant whose crawler it has blocked. If a crawler never reads your services page, it has no way to know you exist, however good your actual service is.

It's worth being honest about the scale of what you'd be protecting versus giving up, using real measurements rather than assumption. Our breakdown of crawl-to-refer data covers Cloudflare's finding that AI crawlers currently send back relatively few visitors per crawl compared to how often they crawl — so the referral traffic you'd be "protecting" by blocking is, for most businesses, smaller than the visibility you'd be giving up in AI assistants' answers.

The cases where the answer is genuinely clear

A handful of situations resolve cleanly, without much agonizing. A business whose entire value is proprietary, paywalled research or analysis has a clear case to block training crawlers on that specific content — it's the product, not the marketing for the product. A local service business with nothing behind a paywall — a dentist, a plumber, a restaurant — has a clear case to allow the major crawlers broadly, since there's genuinely little to protect and real visibility to gain.

Most businesses sit somewhere in between, with some content that's purely promotional and some that's closer to the product itself — which is exactly the situation our robots.txt decision framework is built to work through, path by path rather than as one blanket choice.

What this decision is not about

One thing worth ruling out explicitly: this decision has nothing to do with your Google Search ranking. Google's own documentation states directly that its AI-training control, Google-Extended, "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" — and every other major operator separates its training crawler from the one that affects how it presents or cites you. Making this decision out of ranking fear, in either direction, means making it on the wrong basis entirely.

It's also worth revisiting this decision periodically rather than treating it as permanent. A paywalled publisher that opens a promotional section, or a SaaS company that moves documentation from paid to free, changes which side of this tradeoff applies to that content — the right robots.txt policy follows what the content actually is today, not a choice made once and forgotten.

FAQ