Paywalls, Logins and AI Crawlers: What to Keep Open

The basic mechanics: no special door for bots

A crawler visiting a page behind a required login encounters the same sign-in screen a human non-subscriber does. It has no username or password to offer, so it sees exactly what an anonymous, logged-out visitor sees — which, behind a true login wall, is nothing of the actual content.

This is true regardless of how well-documented or well-behaved the crawler is. OpenAI's GPTBot, Anthropic's ClaudeBot and PerplexityBot are all documented as respecting robots.txt and crawling openly — none of them describe a mechanism for authenticating into a gated area, because that isn't what a crawler is built to do.

The one nuanced exception: Google Search and paywalls

Google Search is a partial exception worth understanding on its own, because it shows this problem has been solved before, just not yet by the AI crawlers. Google documents a system called flexible sampling: a publisher adds specific structured data marking an article as paywalled, and in exchange, Googlebot is permitted to read and index the full article even though an anonymous visitor would hit a paywall — as long as the publisher also shows anonymous visitors some reasonable number of free articles.

This is a real, documented accommodation, but it's specific to Google Search indexing, built through a specific structured-data agreement. As of their own current documentation, OpenAI, Anthropic and Perplexity have not published an equivalent system for their crawlers — so today, the safest assumption for AI-crawler purposes is that paywalled content stays invisible to them, full stop.

Working through the decision by business type

A SaaS company often has two distinct tiers already: public marketing pages and documentation versus an in-app experience or premium help content reserved for paying customers. The marketing pages and public docs are the natural candidates to keep open — they're what would earn a citation and drive a new customer to sign up in the first place. The in-app content can reasonably stay behind its existing login, since it's not meant to attract new customers anyway.

A media or publishing business faces the sharpest version of this tension, since the content itself is frequently the product. A reasonable middle path, mirroring the logic behind Google's flexible sampling even without an equivalent AI mechanism: keep a meaningful set of articles genuinely open — not just a headline and a paywall — so there's real substance for both human readers and crawlers to find, while reserving the deepest, most resource-intensive reporting for subscribers.

A local service business — a dentist, a plumber, a restaurant — rarely has a good reason to put any of its core informational content behind a login at all. There's typically nothing there worth protecting from a business perspective, and everything to gain by keeping it fully open to both crawlers and human visitors.

What to actually do about it

Start by listing which of your pages currently require a login or sit behind a paywall, then ask honestly: is this a page I'd want an AI assistant to cite? If yes, it needs to move outside the gate, or you need to accept that it will never be part of what an AI assistant can say about you. There's no third option, short of building the kind of structured-data accommodation Google has for Search — which, as of today, doesn't yet exist for AI crawlers.

FAQ