AI Crawlers Are Infrastructure Now — Here's How to Think About Them
Retrieval crawlers now determine visibility in AI answers—manage them separately from training bots.

AI crawlers now form part of the infrastructure that connects a website to AI-generated answers. For technical marketing teams, the useful question is not simply which bots to block, but which jobs to permit and what those bots can extract. Teams should treat training and retrieval separately, make content easy to cite, and check whether that access gets the site into answers.
Why the crawler class a bot belongs to is the only decision variable that matters
A noisy crawl report becomes a security ticket, followed by a request to “do something about AI bots.” The request sounds straightforward. Yet a single rule for every AI crawler collapses separate business decisions about model training, search visibility, and live research into one response. Request volume shows how much traffic a bot creates, but not what the business loses by refusing it. Diagnosis depends on what job the bot performs.
Training crawlers gather content for model development, while retrieval crawlers build the indexes assistants query at answer time. User-triggered fetchers do something different again. They retrieve a page because a person is researching right now, so blocking them can interrupt an active prospect’s page read. A fourth group, control tokens, has no incoming crawlers. Google-Extended and Applebot-Extended tell operators how data already collected by other bots may be used.
These distinctions expose an asymmetry in the traffic. Cloudflare Radar shows training as the dominant bucket by request count, with search-purpose crawling accounting for a much smaller share. The busiest class is therefore not the one most directly responsible for present-day answer visibility. Blocking a retrieval crawler resembles blocking Googlebot in 2010, whereas blocking a training crawler addresses a different use of content. A volume-led response can target the wrong business function.
What the named bots actually do, operator by operator
OpenAI separates its main jobs through independent robots.txt tokens. GPTBot handles training, and blocking it has no cost to how often a site appears in answers today. OAI-SearchBot handles retrieval. OpenAI’s crawler documentation says sites that opt out “will not be shown in ChatGPT search answers, though can still appear as navigational links.” ChatGPT-User fetches pages for ChatGPT, Custom GPTs, and GPT Actions, while OAI-AdsBot validates submitted ChatGPT ad landing pages and does not collect data for training.
Anthropic separates training, retrieval, and user-triggered access through ClaudeBot, Claude-SearchBot, and Claude-User. Anthropic publishes a combined list of IPv4 prefixes, without a per-bot breakdown, so that list does not itself distinguish the three jobs. Perplexity likewise separates PerplexityBot, which indexes content for cited answers, from Perplexity-User. Perplexity documents that the former does not crawl for AI foundation models and the latter generally does not follow robots.txt.
Googlebot is the highest-stakes entry in this inventory. Google uses it for Search retrieval and to gate eligibility for supporting links in AI Overviews and AI Mode. GoogleOther is a generic crawler not tied to a specific product, while Google-Agent and Google-GeminiNotebook are user-triggered fetchers. Google-Extended is a control token rather than a crawler, and its effect on Gemini requires a separate policy decision.
Meta’s meta-externalagent supports training and product indexing, while meta-externalfetcher handles user-requested fetches. Meta documents that the fetcher may bypass robots.txt. Meta-webindexer belongs to the retrieval class, as do Bingbot and MistralAI-Index, while MistralAI-Training gathers training data. Applebot indexes for Siri, Spotlight, Safari, and Apple Intelligence, while Applebot-Extended controls whether Applebot data may train Apple models. Common Crawl’s CCBot has no direct retrieval role but feeds many training sets, giving its training access a broader downstream effect.
IP verification deserves its own distinction here. The supplied documentation points to Anthropic’s published list but does not say whether every other named bot has a published list. That documentation gap means a WAF policy cannot assume that every name has an equivalent verification mechanism, or that a shared IP list identifies a particular crawler’s purpose.
How the structural shift in crawl-purpose mix changes the stakes for retrieval
Search-purpose crawling now makes up more of what AI crawlers do, so mistakes in retrieval policy cost more. Cloudflare Radar data through Q3 2026 shows that share is roughly six times what it was a year earlier. Training remains the largest bucket. Its share still fell during the quarter, reaching its lowest point since January in September. Crawl-purpose classification can change, so the exact mix is not a perfectly stable measure.
The direction is more useful than a fixed snapshot. A robots.txt configuration or WAF rule written in early 2025 now faces a different and expanding set of agents. As search-purpose activity grows, a policy built chiefly around the burden of training traffic becomes a poorer fit for the requests arriving today. Static rules age badly when the jobs behind the traffic change.
The split-policy framework that follows from the taxonomy
For teams prioritizing visibility, the practical response is a split policy rather than “allow all” or “block all AI bots.” Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, and Applebot as retrieval crawlers. Their access supports citation eligibility. Treat GPTBot, ClaudeBot, meta-externalagent, CCBot, and MistralAI-Training as separate training decisions, where blocking is a defensible intellectual-property or licensing choice without an immediate visibility cost. A deep training crawl can use a lot of bandwidth, but that cost alone is no reason to block retrieval too.
Allow ChatGPT-User, Claude-User, Perplexity-User, and meta-externalfetcher at the CDN or WAF layer, using published IP ranges where available. OpenAI’s documentation says “robots.txt rules may not apply” to ChatGPT-User because a user initiates the request. Other fetchers document similar limits, making robots.txt an unreliable place to manage this class. Bytespider also belongs at the WAF layer because of its documented history of ignoring robots.txt.
Google-Extended needs a policy decision of its own. Opting out of Gemini grounding reduces eligibility for Gemini app answers while leaving Google Search intact. That is not a clean training-only switch. A team can reasonably choose the restriction, but it must recognize the visibility tradeoff rather than assume that every AI-related opt-out leaves retrieval untouched.
The remaining limit is coverage of the policy itself. No wildcard reliably identifies AI bots as a class, so specific bots must be named individually. A newly introduced, unlisted bot has no individually stated rule. Teams need to keep the split in place as bots come and go, updating the file so it keeps up with the policy. Robots.txt and the WAF serve different purposes within that work.
Why crawler access is necessary but not sufficient — what retrieval infrastructure feeds
The crawler can reach the content, but reachability does not make a passage useful enough to select for an answer. Generative engine optimization, or GEO, addresses that next step by structuring content and building authority so AI platforms select, cite, and surface a brand within synthesized answers. Traditional search aims for a ranked link. GEO depends on retrieval access but asks what the system can use after it arrives.
Peichun Hua and Yunming Xiao’s research on a Semantics Delivery Network describes the mismatch behind this task. Search services return URLs and snippets ranked by how relevant they are in general, while CDNs cache URL-addressed objects without knowing which passage an agent needs. LLMs instead consume short, semantically coherent passages selected for their usefulness downstream. The unit of use is smaller than the page. A page can therefore be available to retrieval without presenting its knowledge in a form suited to an answer.
Definition paragraphs, numbered lists, comparison tables, and FAQ blocks give systems extractable chunks. Comparison pages built around tables outperform prose-only articles containing the same underlying facts. The facts alone are not the whole product. Prose-heavy AI-generated articles can be indexed yet rarely cited when they lack this structure. Publishing volume is not a substitute for content architecture. Access opens the route, while the page’s organization determines what is ready to travel through it.
What content architecture makes a site legible to retrieval systems
A technical marketing team needs to make each passage that an answer could use clear enough to read without the full page. A short, semantically dense passage should answer a specific question. Definitions work best as definition blocks, comparisons as tables, and processes as numbered lists. These formats express relationships. The aim is not to shorten everything, but to give an answer a clear boundary and enough context to stand on its own.
Claims also need named, verifiable sources attached to them. Topical relevance is not enough when systems weigh the citability of the source itself. People still need to check the work, especially when AI tools help write large numbers of drafts. A fluent paragraph can conceal an unsupported claim just as easily as an accurate one. Extraction-friendly formatting cannot repair that defect.
Mark Russinovich, Ram Shankar Siva Kumar, and Ahmed Salem’s July 2026 Microsoft preprint examined hallucinated citations in tens of thousands of accepted papers from ICLR, ICML, NeurIPS, and USENIX Security. It found made-up references that were now part of the permanent scholarly record. The papers with the most failures reflected workflow breakdowns rather than fraud, with LLM-generated bibliography entries left unverified. Nobody in the pipeline owned the check. For a marketing team, the parallel risk is a claim or citation reaching publication because the workflow has no review gate.
That check needs an owner. Human review protects the credibility of the source that retrieval systems are being asked to use. Freshness adds another requirement because answer engines weigh both recency and coverage. A site publishing quarterly on a topic competes poorly against one with a publishing cadence and standing watchers that draft when relevant developments occur. Those drafts still need the same review gate, so speed does not become a route around evidence.
How to know whether the retrieval infrastructure is actually working — measurement as the closing loop
Crawler access and content architecture create eligibility. Measurement must establish whether engines actually name and cite the brand. A single answer cannot settle that question. Citation share changes from engine to engine and over time as models rebuild answers based on how fresh, varied, and wide-ranging the content is. A prominent appearance in one engine today may coexist with near-absence elsewhere or disappear from the next day’s response.
Share of voice shows how often a brand appears, while teams need to track citations and unlinked mentions separately. Sentiment shows how the answer talks about the brand. Source-domain tracking shows whose content the answer uses, and prompt-level tracking shows which questions get the brand into an answer. Together, these signals distinguish a broad visibility gap from a narrower weakness in the questions a site can answer.
SE Visible from SE Ranking tracks brand appearance across ChatGPT, Gemini, Google AI Mode, Google AI Overviews, and Perplexity, covering mentions, share of voice, sentiment, citations, and source domains. Semrush AI Toolkit monitors citations across the same engines and surfaces the prompts that trigger a brand, as well as competitors appearing in the same answers. Ahrefs Brand Radar covers AI Overviews, AI Mode, ChatGPT, Perplexity, Gemini, and Copilot, reporting mentions and citations separately. These tools make AI visibility a measurable category. Their coverage also makes the choice of engines an explicit part of measurement.
For teams managing this infrastructure, tracking should run on a schedule. Measuring again over time shows whether the access policy and the work on content get the site into answers as intended. AI crawlers are infrastructure because they connect a site’s knowledge to the answers people receive. Configuring access is the entry fee, extractable content is the substance, and measured citation presence closes the loop.


