A 90-Day Plan to Turn AI Crawlers Into Customers

Weeks 1–2: find out what's actually true

Before changing anything, establish the real facts. Check your server logs for claimed AI-crawler visits, then verify they're genuine by matching the IP address against the real published lists OpenAI, Anthropic and Perplexity each maintain — the full verification walkthrough is here — since a name in your logs alone proves nothing.

At the same time, check what those crawlers can technically see on your key pages. View your raw page source and confirm your core facts — services, prices, location — appear in the text itself, not just in what a browser eventually displays after JavaScript runs.

Weeks 3–6: fix access and structure

With real facts in hand, fix what's actually broken. Work through the robots.txt decision framework — three questions, not a list of bots to memorize — to set a deliberate policy instead of an inherited default.

Then address the technical gaps the first phase surfaced: move key facts into your server's first response, fix any content trapped behind accordions or tabs that only load on click, and review the full technical checklist for anything else standing between your content and a crawler that can't run JavaScript. If any of your content sits behind a login or paywall, decide deliberately what stays open rather than leaving it to a default.

Weeks 7–10: build content that actually earns a citation

Being readable isn't the same as being worth citing. This phase is about the gap between crawled and cited: replacing generic language with specific, verifiable claims, making sure your business's identity and credentials are clearly stated, and checking whether independent sources — reviews, directories, outside mentions — corroborate what your own site says about you.

This is also the point to make a clear-eyed decision about the real tradeoffs of blocking versus allowing specific crawlers, now informed by actual data about your own site rather than a general fear either way.

Weeks 11–12: measure, and set realistic expectations

Close the loop by checking what actually changed. Revisit your server logs for real (verified) crawler visits, and periodically test whether AI assistants now mention your business for the searches that matter to you.

Keep your expectations grounded in what the real traffic numbers actually look like — Vercel measured AI crawler volume at a fraction of Googlebot's, and Cloudflare found most of it serves training rather than sending you a specific visitor (the full breakdown is here). Progress here is closer to slow compounding than an overnight surge, and that's normal, not a sign of failure.

One myth to leave behind before you start

None of this plan should be driven by fear of losing Google rankings. Google's own documentation states plainly that its AI-training control, Google-Extended, "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" — and every other major operator separates training from search-visibility the same way. Make every decision in this plan on its own merits, not out of ranking anxiety.

What this looks like for one real kind of business

Picture a family-run veterinary clinic working through this plan. In weeks 1–2, the owner checks her server logs, finds a handful of requests claiming to be GPTBot, and confirms two of them are genuine by matching the IP addresses against OpenAI's published list — the rest turn out to be unrelated traffic that simply used the same name.

In weeks 3–6, she discovers her "Services" page loads its list of treatments through a JavaScript tab system, invisible to a non-rendering crawler, and has her web developer move that list into the page's initial HTML. She also writes a simple, deliberate robots.txt policy: allow the major crawlers broadly, since nothing on her site is paywalled and everything to gain from being found.

In weeks 7–10, she rewrites her About page to replace "compassionate, quality care for your pets" with specific, verifiable facts: which animals the clinic treats, how long it's operated in its neighborhood, and her own veterinary credentials — the kind of specific claim an assistant can actually cite. By weeks 11–12, she starts periodically asking a couple of AI assistants herself what they say about veterinary care in her area, and over a few months, she begins to see her clinic mentioned where it wasn't before.

FAQ