How AI Crawlers Actually Read Your Website
By Zakaria Reziki
CEO — Growli · September 30, 2026 · 7 min read
Drafted with AI assistance under editorial standards set by Zakaria Reziki, then published after automated sourcing and quality checks.
Most AI crawlers read the raw HTML your website's server sends back, and nothing else — if your page needs JavaScript to fill in the text, prices or services a visitor sees, that content is often invisible to them. This is not a rumor or a guess: it is what the companies that build these crawlers, and the infrastructure providers whose servers get hit by them millions of times a month, have measured and published themselves.
For a search engine, this distinction barely matters anymore — Google has spent over a decade building a rendering pipeline that lets Googlebot see a page the way a visitor does. For an AI assistant deciding whether to mention your business, it matters enormously, because the leading AI crawlers have not built that same pipeline. What follows is a plain explanation of the difference, who the actual crawlers are, and what a business owner can do about it this week.
Fetching versus rendering: the difference that decides what a crawler sees
Every website visit starts the same way: a request goes to your server, and the server sends back a response. That response is HTML — the raw skeleton of the page. A modern website often ships that skeleton nearly empty on purpose, then uses JavaScript — small programs that run inside the visitor's browser — to fill it in afterward with the real text, prices, and images.
A human visitor's browser runs that JavaScript automatically, so they never notice the gap. "Rendering" is the technical word for that extra step: running the JavaScript and building the finished page. "Fetching" is just grabbing the raw HTML and stopping there. Most AI crawlers fetch. They do not render.
Picture a plumber's website where the homepage loads instantly with a logo and a photo, but the actual list of services — drain cleaning, water heater repair, emergency calls — only appears a second later, filled in by JavaScript after the page loads. A visitor's browser does that filling-in so smoothly they never notice two steps happened. An AI crawler that only fetches sees the empty shell and stops there. The services list, to that crawler, does not exist.
The bots, one by one, and what each one is actually for
"AI crawler" is not one bot — it is a handful of companies, each running several bots for different jobs. Knowing which is which matters, because blocking one doesn't block the others.
OpenAI documents four separate bots: GPTBot gathers content that may be used to train its models. OAI-SearchBot crawls to decide what to surface when someone searches inside ChatGPT. OAI-AdsBot checks the safety of a specific ad landing page. ChatGPT-User is different from all three — it only visits a page in real time because an actual person asked ChatGPT a direct question, and OpenAI's own documentation notes that "robots.txt rules may not apply" to it, since a person explicitly requested that fetch.
Anthropic runs a similar split: ClaudeBot collects content "to enhance the utility and safety of our generative AI models," Claude-SearchBot crawls to improve search-style results inside Claude, and Claude-User fetches a page only when a person's question sends it there. Perplexity runs the same pattern: PerplexityBot crawls to surface and link websites, while Perplexity-User fetches a page only in response to a person's question.
None of these bots render JavaScript. All of them, according to the companies' own documentation above, are meant to respect robots.txt — except the user-triggered ones, which exist specifically because a real person is waiting on the other end.
The bots by name
One company, several bots, different jobs
GPTBot, ClaudeBot
Crawl broadly to gather training content. Respect robots.txt.
OAI-SearchBot, Claude-SearchBot, PerplexityBot
Crawl to power search-style answers inside the assistant. Respect robots.txt.
ChatGPT-User, Claude-User, Perplexity-User
Fetch one page in real time because a person asked a direct question. May not follow robots.txt.
Googlebot is the outlier, and it's worth knowing why
If AI crawlers don't render JavaScript, why does this feel unfamiliar? Because Googlebot does. Google documents that Googlebot exists "to find information for building Google's search indexes" and, to do that well, Google built a full rendering pipeline over many years — essentially running a real browser at enormous scale for every page it wants to understand fully.
That is a genuinely expensive piece of infrastructure, and right now, the AI companies have not built their own version of it for their crawlers. Vercel's own analysis states it plainly: "none of the major AI crawlers currently render JavaScript. This includes OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ByteDance (Bytespider), Perplexity (PerplexityBot)." The same analysis found these crawlers do sometimes fetch a site's JavaScript files — ChatGPT's bots did so in about 11.5% of requests, Claude's in about 23.8% — but fetching the file is not the same as running it. They still don't execute it. In that same period, Vercel measured 4.5B monthly Googlebot requests against 569M from GPTBot and 370M from Claude's crawlers — a fraction of the volume.
That gap won't necessarily last forever. But for a business owner deciding what to do with a website today, it's the reality right now, not a permanent law of nature — worth revisiting occasionally, not worth losing sleep over.
Vercel, Dec 2024
Monthly requests: AI crawlers vs. Googlebot
Googlebot
4.5B
GPTBot
569M
Claude (crawlers)
370M
Source: Vercel
What this actually costs a real website
Go back to the plumber's site. If the services list, the service-area map, and the customer reviews all load in after the initial page through JavaScript, here's what happens when someone asks an AI assistant "who does emergency plumbing near me": the assistant's crawler fetched that page at some point, saw an almost-empty shell, and moved on. It never learned what services this plumber actually offers.
This isn't a small technical footnote — it is, in plain terms, the difference between a business existing in an AI assistant's picture of the world and not existing at all. A rival plumber whose site loads its services directly in that first server response is simply more visible to be recommended, with nothing else about their business being any better.
How to check what your own site actually shows a crawler
You don't need to be technical to check this yourself, and it takes about two minutes.
- Right-click your homepage and choose "View Page Source" (or press Ctrl+U / Cmd+Option+U). What you see is close to the raw HTML a crawler fetches. If your services, prices, or key facts aren't in that text, they may not be in what an AI crawler can read either.
- Turn off JavaScript in your browser for a moment (most browsers offer this in their developer tools or settings) and reload your homepage. Whatever disappears is exactly what a non-rendering crawler never sees in the first place.
- Ask whoever built or maintains your site one direct question: "Is our main content — services, prices, location, hours — sent in the first server response, or filled in afterward by JavaScript?" If the honest answer is "filled in afterward," that's the fix to prioritize.
What to do next
None of this requires rebuilding your website. It means making sure the words that matter most — what you do, where you are, what it costs, why someone should pick you — arrive in that very first response from your server, before any JavaScript runs at all. Structured data and a clean, crawler-friendly page setup build on exactly this foundation, and Growli checks whether AI assistants can actually see and recommend your business today, not just whether your site looks right in a browser.
See what AI says about your business
Growli measures your share of AI answers across ChatGPT, Gemini, Claude and Perplexity — and turns every gap into prioritized actions.
Get StartedFAQ
Not among the major ones tested. Vercel's analysis of real crawler traffic found that OpenAI's, Anthropic's, Meta's, ByteDance's and Perplexity's crawlers all fetch raw HTML only. Google's Search infrastructure is the clear exception because Googlebot shares Google's existing rendering pipeline.
No — this is a separate question from Google ranking, since Googlebot already renders JavaScript fine. Making your content available without JavaScript is purely about the AI crawlers that don't render it; it won't change how Google treats your site.
Running a full browser to render every page at the scale these companies crawl is expensive infrastructure that Google spent years building specifically for Search. It's a reasonable guess that cost and engineering priority are part of why AI companies haven't built the same thing yet, though none of them has stated that publicly as the reason.
Single-page applications — sites built entirely in frameworks like React or Vue, where even the homepage's basic text is generated by JavaScript — carry the most risk, since a non-rendering crawler may see almost nothing at all. A traditional site with a few JavaScript-loaded sections carries a smaller, more contained risk.
No. This is about what a crawler can technically read when it visits, not whether you've told it to stay away. A site can be fully open to every crawler and still be invisible to them if its content only appears after JavaScript runs.