How to Verify Real AI Crawler Visits in Your Server Logs

Why the name in your logs isn't enough on its own

Picture a dentist's office whose website logs show fifty visits from something calling itself "GPTBot" last month. It feels like proof that OpenAI is reading the site. But a user-agent string is entirely self-reported — like a visitor at the front desk simply saying their name out loud, with nobody checking an ID. A different program, built by anyone, can say the exact same name.

This matters for two practical reasons: if you're deciding whether to block or allow a crawler, you want to know you're actually controlling the real one, not a copycat. And if you're trying to understand how much real AI-crawler traffic you get, an inflated or faked count gives you the wrong picture entirely.

The real verification method: published IP lists

Each major operator solves this the same way: they publish a machine-readable file listing the actual IP address ranges their crawlers send requests from. A request is only genuinely that company's crawler if both the user-agent name and the source IP address match what the company itself publishes.

OpenAI publishes separate verification files for each of its bots — GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User each have their own list. Anthropic publishes one list covering its crawlers at claude.com/crawling/bots.json. Perplexity publishes files for both of its bots — PerplexityBot and Perplexity-User — and its own documentation recommends checking both the user-agent name and the IP address together for a reliable result.

The four-step check, in plain terms

You don't need to be a developer to run this check, though having one help with the technical steps makes it faster.

• Find the claimed visits in your logs. Search your server's access logs for the bot name you're checking — "GPTBot," "ClaudeBot," or "PerplexityBot" — and note the IP address each request came from.
• Open the company's published IP list. Each is a small, plain text file at the addresses above, meant to be read by software but perfectly readable by a person too.
• Compare the IP address, not just the name. If the IP address from your logs appears in the company's published range, the visit is real. If it doesn't, something claimed a name it doesn't own.
• Recheck periodically. Companies update these ranges as their infrastructure changes, so a one-time check goes stale. Most site owners only need to recheck every few months, or whenever something about their crawler traffic looks unusual.

What this actually protects you from

This isn't about paranoia — it's about making an informed decision instead of an assumed one. A business owner who decides to allow GPTBot because they want AI assistants to learn about their products should know that the traffic they're allowing is actually GPTBot, not an unrelated scraper borrowing a trusted name to get past a simple block.

The same check matters in the other direction: a business that blocks GPTBot in robots.txt is trusting that legitimate, well-behaved crawlers respect that instruction — which OpenAI's own documentation says GPTBot does. A faked visit wearing GPTBot's name wouldn't be bound by that same honesty, which is one more reason the IP check matters if you ever see traffic that looks off. Once you've confirmed who's really visiting, our guide to how these crawlers read your site and the facts on whether blocking any of them affects your Google ranking cover the decisions that come next.

FAQ