The Two-Class Crawler Problem: Why Most Teams Are Getting Bot Governance Wrong
Separating training and retrieval crawlers prevents accidental citation bans.

Training crawlers collect material for model weights, while retrieval crawlers help engines find sources for answers. Treating them as one class can turn a decision about training rights into an accidental ban on citations. Separating those decisions is the technical prerequisite for AI visibility, though earning citations then requires useful content, authority and measurement.
The functional difference between training crawlers and retrieval crawlers that most teams never learn
Training scrapers collect content to build or update model weights. Retrieval indexers fetch live content to ground answers at query time. These are different uses of the same material. One concerns what goes into a model, while the other concerns what an engine can find when answering a question. A policy about the first use does not, by itself, express a position on the second.
The crawler names make that distinction concrete. Anthropic uses ClaudeBot for training, while Claude-SearchBot and Claude-User serve retrieval and citation functions. The shared company name does not make their jobs interchangeable. Their roles give a site owner separate choices about how content is used, rather than a single choice about whether to admit Anthropic.
The pattern extends beyond that company. PerplexityBot serves search retrieval and citation, while Google-Extended concerns Gemini training, separate from Search indexing. Googlebot supports grounding for AI Overviews and AI Mode, and Bingbot supports Copilot grounding. A crawler’s function matters more than whether it belongs to an AI company.
How teams end up blocking citation crawlers while thinking they are protecting their content
A wildcard block aimed at “AI scrapers” can catch OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot together with the training crawlers it was meant to stop. The intention may be narrow, but the rule is broad. It treats the identity of the visitor as a substitute for the purpose of the visit.
The consequences are not symmetrical. Blocking a training crawler is a policy choice about what goes into future model weights, though how much that choice changes is open to debate. Blocking a retrieval crawler cuts off the site from that crawler’s citation pool immediately. Anthropic’s documentation states that blocking Claude-SearchBot “may reduce your site's visibility and accuracy in user search results,” while blocking Claude-User “may reduce your site's visibility for user-directed web search.”
SEO and security plugins offering an “AI bot protection” toggle have issued wildcard blocks without distinguishing crawler class. A toggle compresses several policy decisions into one. Rules at the CDN layer can likewise stand between a retrieval crawler and a page, so robots.txt alone does not describe the whole access policy. Ordinary monitoring may not reveal the lost citation opportunity, which helps explain why technically competent teams can overlook it.
How widespread the misconfiguration actually is, and what the data reveals about where policies are absent entirely
In 2026, most sites have no AI-specific rules. Their behavior is instead determined by wildcard or legacy rules written for other purposes. Paul Bouchaud and Pedro Ramaciotti’s Web Crawler Restrictions, AI Training Datasets & Political Biases found that roughly a quarter of the top thousand most-visited sites restrict AI crawlers. Across the top million sites, that share is much lower.
News outlets restrict crawlers at much higher rates than other categories, and outlets with high factual reporting restrict at the highest rates. Neutral, high-factual outlets do the most blocking, while hyperpartisan and low-factual-reporting sites do the least. The authors flag a risk to the quality and representativeness of training data when the sources most willing to admit crawlers differ systematically from those keeping them out.
Among sites with explicit policies, retrieval receives less attention. A parse of robots.txt files found that every vendor’s citation crawler was named less often than its training crawler. Naming frequency does not prove that every unnamed retrieval crawler is blocked. A missing retrieval rule leaves its treatment dependent on whatever broader rules apply.
There is a further complication, but it is a different one. Steven Seiden and colleagues’ Identifying AI Web Scrapers Using Canary Tokens documents scrapers operating outside stated policies. Compliance is voluntary and imperfect. Robots.txt expresses instructions rather than inherently enforcing them, so a written restriction and an effective access control are not equivalent. That limitation prevents the policy file from being treated as a complete account of crawler behavior.
Why getting crawler access right is a necessary condition for citation, not a sufficient one
Removing a block from robots.txt or CDN rules restores eligibility. It does not establish that a page deserves citation. Once a retrieval crawler can reach the content, the engine still has to choose it over other available sources. That choice depends on topical authority, structural extractability and third-party corroboration.
This is the strongest objection to an access-focused strategy. A site can admit every retrieval crawler and still never appear in AI answers because it lacks authority on the relevant topics. Accessible domains do not receive citations evenly. A small number of high-authority domains account for a disproportionate share of consolidated citation volume across engines.
The canary-token study shows that the connections between scrapers and the chatbots they feed are not always publicly known. Permission does not make citation routing transparent. An allowed crawler’s visit cannot, on its own, explain which engine will use the material or whether the material will appear in an answer.
So does this weaken the case for fixing crawler rules? It limits the promise. Access is a blocked-or-not condition, while citation probability varies with the strength of the content and its supporting authority. The lesson for a team seeking visibility is to treat crawler configuration as a prerequisite, then pursue the content work that permission alone cannot do.
What content architecture gives retrieval crawlers something worth citing once they can reach you
That content work begins with pages a retrieval system can discover, interpret and use as sources. Good writing is part of the task, but not the whole task. A page also needs a clear topical role and a structure from which useful information can be extracted. Third-party corroboration strengthens the authority behind that information.
Discovery comes before extraction. A sitemap line in robots.txt is the cheapest and most direct signal a site can send about which pages exist and are worth fetching. Most sites with AI-access problems also lack a correctly referenced sitemap, compounding the discovery failure. For teams correcting access, that makes the sitemap part of the same practical work.
The policy vocabulary is also evolving beyond robots.txt. The llms.txt proposal and the ai.txt framework address gaps in how sites express meaning and permitted uses to AI systems. Yuekang Li and colleagues’ ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet proposes element-level rules that could permit retrieval and citation while disallowing training or summarization at the paragraph or image level. The framework supports XML-based programmatic enforcement and natural language prompt integration. Adoption remains early, so these approaches are directional rather than fully standardized replacements for existing controls.
Freshness adds another variable to the architecture. Retrieval crawlers fetching at query time weight recency differently by engine, so an update cadence influences some engines more than others. Teams should connect maintenance to the engines they hope to reach, accounting for the value each engine places on freshness. Even with new policy files, sites need authoritative pages that engines can extract information from and that are still useful when an engine needs them.
Why the citation surface extends well beyond the brand domain and what that means for governance
Once access and content quality are understood, the next question is where that content needs to exist. The brand domain is only one surface. The majority of AI citations across answer engines come from a small set of high-authority third-party platforms, including Reddit, Wikipedia, YouTube, LinkedIn, Forbes and roughly ten more similar domains. A brand’s robots.txt governs none of those platforms.
That changes the scope of the strategy. A brand publishing exclusively on its own domain can make its pages accessible and authoritative while leaving much of its potential citation surface unaddressed. If an engine tends to cite third-party sources, better rules on the brand’s site do not create a presence in those sources.
A ground-site and phantom-site model offers a practical way to think about this division. These are practitioner labels. The ground site is the primary domain, made accessible to retrieval crawlers, structurally extractable and authoritative on target topics. The phantom surface is the distributed presence on high-authority third-party domains that engines weight. It sits outside the brand’s robots.txt, even when it contributes to answers about the brand.
By far most citations come from editorial and third-party coverage, with a much smaller share coming from brand-owned pages. A brand’s distributed presence includes both coverage and material it publishes elsewhere. For marketing and content teams, the lesson is to pursue direct citation and third-party authority as related work, while recognizing the limits of their control.
How to measure whether crawler configuration and content changes are actually producing AI citations
Measurement has to follow that wider surface and distinguish engines rather than collapse them into one AI visibility score. Most teams lack that measurement layer. Perplexity, Claude, ChatGPT and Gemini cite meaningfully different numbers of unique domains per answer, and overlap between the domains cited by ChatGPT and Perplexity is low. The same brand’s citation volume can also vary by orders of magnitude between engines. Teams should track whether each engine names and cites the brand, with separate goals for cross-engine consistency and the total number of mentions. A result on one engine cannot stand in for visibility across the others.
Teams correcting crawler policy should assess robots.txt and CDN rules together, separating intended permission from actual compliance and separating both from citation outcomes. Engine-level citation tracking across engines requires purpose-built tooling beyond the capabilities of standard analytics and repurposed SEO dashboards. The useful test is whether access changes remove a barrier and content changes improve selection, without confusing those two results. That distinction turns bot governance from a blanket defensive setting into a deliberate choice about training rights and participation in AI answers.


