AI Crawler Traffic vs. Googlebot: What the Numbers Really Look Like

How often AI crawlers actually visit, compared to Googlebot

Vercel, whose servers host a large share of the modern web, published its own traffic analysis covering a full month across its network. Googlebot generated 4.5 billion requests in that period. GPTBot generated 569 million — about one-eighth the volume. Claude's crawlers generated 370 million, smaller still.

Put plainly: for every visit Googlebot makes to the web Vercel hosts, GPTBot makes roughly one-eighth of a visit, and Claude's crawlers make even less. That's not a rounding error — it reflects how much longer Google has been building and running Search infrastructure at scale.

Crawling and sending visitors back are two very different things

Raw crawl volume answers "how often does this bot visit," but a business owner usually cares about a different question: "does this send me customers?" Cloudflare, which sits in front of roughly a fifth of all web traffic by its own account, measured both sides of that question for one week and published what it calls the crawl-to-refer ratio — how many times a company's bots crawled a page for every one time they sent a real visitor there by link.

The numbers are stark. Cloudflare found OpenAI's overall ratio near 887 crawls for every 1 referred visitor. Anthropic's was far higher still — "nearly 50,000:1" in Cloudflare's measurement. Perplexity, whose product is built around answering with links, had a far tighter ratio of about 118 to 1 — still lopsided, but the closest of the three to sending visitors back for what it crawls.

Why the ratio is so lopsided: most crawling is for training, not answers

The reason the crawl-to-refer gap is so wide traces back to what most of this crawling is actually for. Cloudflare's same research found that "training traffic, responsible for nearly 80% of the crawling from AI bots," dwarfs the traffic used to answer a specific question in real time — the kind of traffic most likely to include a link back to your site.

Training crawls gather content to build or improve a model generally — they have no direct reason to send a visitor anywhere, because no specific person asked a specific question. A user-triggered fetch, like ChatGPT-User or Perplexity-User visiting a page because someone asked a real question, is the traffic most likely to translate into an actual visitor, and it's a small share of the total crawling that happens.

What this means for a real business

Picture a local accountant during tax season. Her website might see thousands of Googlebot visits a month and a handful of AI-crawler visits by comparison — and that's expected, not a sign her site is failing with AI assistants. What actually matters for her is less about volume and more about two narrower questions: is she being crawled at all by the assistants her potential clients use, and when someone does ask one of those assistants a direct question, does it know enough about her to answer with her name in it.

Chasing AI-crawler visit counts the way businesses have chased Googlebot activity for decades isn't the right instinct here — the volume simply isn't built the same way yet. Our guide to diagnosing whether you're crawled but not cited goes into what actually predicts a mention, which matters more than raw visit counts.

FAQ