Crawled but Never Cited: Why AI Assistants Ignore Some Sites

Why "we were crawled" doesn't mean "we'll be cited"

A crawler visit only confirms that a bot successfully fetched your page — it's the mechanical first step, not the outcome. What happens after that visit — whether the content is specific enough, trustworthy enough, and distinct enough to be worth citing — is a separate judgment the AI system makes later, often not tied to any single visit at all.

Think of it like a journalist who reads dozens of sources while researching a story but only quotes two or three of them directly. Being read is necessary to be quoted. It is nowhere near sufficient — Cloudflare's own measurement of AI crawler behavior shows just how much crawling happens generally, relative to any specific outcome for any specific site.

The most common reason: content that says nothing specific

Picture two dentists' "About Us" pages. One says "We provide quality dental care with a friendly team in a comfortable environment." The other says "Our office has treated families in the Riverside neighborhood since 2009, specializes in same-day emergency appointments, and our lead dentist completed advanced training in pediatric sedation." An AI assistant answering "which dentist near me handles emergency same-day visits" has something concrete to work with from the second page and nothing usable from the first — even if both pages were crawled identically.

Generic language isn't wrong, exactly — it's just not citable. An assistant generating an answer needs specific claims it can attribute to a specific business, not a paragraph that could describe almost any competitor equally well.

The second reason: missing or weak trust signals

Google's own guidance on what makes content worth ranking describes prioritizing content that demonstrates "experience, expertise, authoritativeness, and trustworthiness" — commonly shortened to E-E-A-T. While that guidance is written for Google Search specifically, the underlying judgment — does this source seem credible enough to rely on — is a reasonable, everyday standard that any system summarizing information for someone would apply, human or automated.

A page with no named author, no clear business identity, and no verifiable specifics reads as less trustworthy than one that plainly states who wrote it, what qualifies them, and where the business is actually located. Our guide to entity consistency covers the identity side of this in depth — a business that's described inconsistently across its own site is harder for any system to trust confidently.

The third reason: no one else says it too

A claim that appears only on your own website carries less weight than the same claim echoed independently elsewhere — a review, a local directory listing, a mention on someone else's blog. This isn't a conspiracy against self-promotion; it's a reasonable caution any information-summarizing system would apply, since a business's own claims about itself are the least independently verified source available. It's also worth remembering that PerplexityBot's own documented purpose is specifically "to surface and link websites" — a system built around citing sources has every reason to prefer a claim it can point to in more than one place.

This is why reviews and outside mentions genuinely matter here, beyond their obvious marketing value — they're corroboration an AI system can reasonably treat as more independent than your own homepage.

A practical audit you can run today

Pull up your own key pages and ask three honest questions about each one.

• Could a competitor's page say the same sentence? If yes, the sentence is too generic to be a citable fact about your specific business.
• Would a stranger reading only this page trust it? Check for a named person, a specific location, a verifiable detail — not just a friendly tone.
• Does anything else on the internet back this up? A review, a directory listing, a mention elsewhere. If the only place a claim appears is your own site, it's the weakest version of that claim.

FAQ