AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and Friends

The bots, one company at a time

Think of robots.txt the way you'd think of a sign on a shop door: "deliveries around back, customers welcome in front." It only works if every visitor reads signs — and it only sorts visitors into groups if you know which sign applies to which one. Here is who is actually knocking.

OpenAI documents four separate bots. GPTBot is the one that gathers text to help train and improve its models — the equivalent of a researcher reading library books to get smarter, not to answer any specific question. OAI-SearchBot is different: it crawls pages so ChatGPT's search feature can find and cite them when a user asks something, closer to how a search engine indexes the web. ChatGPT-User visits a page at the exact moment a person is chatting with ChatGPT and the assistant needs to check something live — think of a shop assistant stepping into the back room to check if an item is in stock, right in front of the customer. OAI-AdsBot checks that pages submitted as ChatGPT ads are safe to send people to. OpenAI documents all four, and their robots.txt behavior, on its developer site.

Anthropic (the company behind Claude) runs three. ClaudeBot gathers pages for model training, the same library-reading role as GPTBot. Claude-SearchBot crawls pages to improve what Claude can find and cite in its search results. Claude-User fetches a specific page the moment a Claude user asks about it — the same "checking in front of the customer" behavior as ChatGPT-User. Anthropic explains all three, and how to control each one, in its help center.

Perplexity keeps it to two. PerplexityBot crawls and indexes pages so Perplexity's answers can link to them — again, the search-indexing role. Perplexity-User fetches a page live, on a user's behalf, the moment someone asks a question that needs it. Perplexity's own documentation names both and explains the difference.

Two jobs, not one: training versus "right now"

Once you see the pattern above, the whole bestiary collapses into two simple categories, and this is the distinction that actually changes what you should do.

Training and search-indexing bots — GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot — visit on their own schedule, independent of any single customer. They are how these companies build up general knowledge and a searchable index of the web over time. Blocking one of these tells that company "don't use my future pages for this purpose" — it is a genuine, meaningful choice, but it takes effect slowly and does not touch anything already learned before you blocked it.

User-triggered bots — ChatGPT-User, Claude-User, Perplexity-User — visit because an actual person, mid-conversation, asked the assistant to look at your page right now: to check current pricing, read today's menu, or confirm your hours. Both OpenAI and Perplexity say plainly that because a person requested the fetch, this kind of visit may not honor robots.txt the way an automated crawl does — the reasoning being that a person asking a direct question deserves the same access they'd get by clicking the link themselves.

This is why blocking "the AI bot" as one undifferentiated thing so often backfires: you can turn off the crawler that would have made your business more discoverable next month, while doing nothing at all to stop a page from being fetched live tomorrow — because that fetch was never yours to block in the first place.

Reading your own server logs to see who actually showed up

You do not need special software to check whether any of this is happening to your site. Every web server keeps an access log — a running list of every visitor and what they looked at — and every crawler identifies itself in a line called the user-agent, the same way a delivery driver might show a badge.

If your site is hosted somewhere with shell access, you can search that log yourself. A command as simple as looking through your access log for the text "GPTBot", "ClaudeBot" or "PerplexityBot" will show you every visit, with the date, the exact page requested, and whether the server let the visit through or blocked it. Most hosting control panels (cPanel, Plesk, and similar) offer the same access log through a point-and-click "raw log" or "visitor log" screen if you'd rather not use the command line.

Two things are worth checking once you find them: how recently a given bot last visited (a training bot that has never visited means your pages are not yet part of that company's training data), and whether the requests came back successful or blocked — a page that returns an error to GPTBot might also be returning that same error to a real customer's browser, and you'd want to know either way.

Writing a robots.txt that helps you instead of hiding you

robots.txt is a plain text file you place at the root of your website (yoursite.com/robots.txt) that lists rules for which automated visitors may read which parts of your site. Google's crawling documentation — written for its own search crawler, but the file format is shared by every crawler operator — is the clearest plain-language explanation of the syntax if you have never opened one.

For almost every local or small business, the safest starting position is the simplest one: allow every crawler to read every public page, and write no Disallow rules at all for AI bots specifically. The upside of being crawled is a chance to be mentioned, compared and recommended; the downside of blocking is being left out of an answer entirely, with no way to know it happened. There is no version of "partially blocked" that makes an assistant describe you more favorably — it can only make you invisible to that one bot.

The two real exceptions are pages you would not want a search engine to show either — a private customer portal, an internal admin page, a draft you have not published — which should already carry a Disallow rule for every crawler, AI or not. And a page that is genuinely misleading if read out of context on its own, without your site's design and disclaimers around it, which is rare, but worth a second look if it applies to you.

If you do decide to block a specific bot, use its exact name — "GPTBot", not "OpenAI", not "AI bots" — because each bot only obeys rules written for its own name. A rule that blocks GPTBot has no effect on OAI-SearchBot, and vice versa; that separation is the whole reason these companies split the bots apart in the first place.

• List each bot you want to address by its exact name, one User-agent line per bot.
• Leave the file empty of AI-specific Disallow rules unless a page is genuinely private.
• Re-check the file after any site migration — a new host sometimes ships a default robots.txt that blocks everything.
• Test the live file at yoursite.com/robots.txt in a browser after publishing; a typo here is silent and easy to miss.

FAQ