Computational Marketing
FeaturesLong read

Why your CDN is the real gatekeeper for AI citation — and what to do about it

CDN and firewall rules can block AI crawlers before robots.txt permissions ever matter.

· 8 min read
Cover illustration for “Why your CDN is the real gatekeeper for AI citation — and what to do about it”
Features · November 4, 2026 · 8 min read · 1,878 words

AI citation starts with access, and your CDN can deny that access before a crawler ever reads robots.txt. That makes CDN and WAF configuration both a marketing concern and a security setting. The response should let retrieval and user-triggered crawlers through every layer while keeping other limits in place, then check whether the pages they can reach earn citations.

Why robots.txt is not the enforcement layer for AI crawler access

A team can configure robots.txt to allow AI crawlers while its infrastructure refuses their requests. The file describes intent. A reputable crawler may honor every instruction in it, but that makes no difference if a web application firewall, or WAF, blocks the request before it reaches the origin server. The crawler never gets to read the permission.

The order explains the apparent contradiction. CDN and WAF rules execute at the edge, before the crawler can evaluate any per-path or per-agent instructions in robots.txt. A site can therefore be allowed in robots.txt and blocked by the WAF at the same time, with the block determining the outcome.

What field measurements show about how widespread silent blocking actually is

A field measurement run on 2026-10-04 across Amsterdam bike-shop home pages found 19 of 79 reachable sites returning 403 to at least one AI crawler while returning 200 to a browser user agent. ClaudeBot was refused on all 19, GPTBot on 14, and PerplexityBot on 5. The mismatch went beyond an omitted permission. On 15 of the 19 sites, robots.txt explicitly allowed the crawler that the server then refused. On 12, even /robots.txt returned a refusal when fetched as GPTBot.

Those results describe only that sample. They do, however, expose a failure that a browser visit or a robots.txt review alone would miss. An earlier Japanese tourism sample showed the same pattern, so a quirk in the hosting region is less likely to explain it. The consistency across unrelated samples on different continents is the stronger signal. It gives teams a reason to test their own properties without assuming that the Amsterdam percentage applies to them.

Why this happens — security defaults that were never designed with AI citation in mind

The Amsterdam measurement found no evidence that the refusals were intentional. Often, the system is doing what it was told to do for a different problem, through security rules that predate AI crawlers or were never reviewed for their effect on citation. A rule against traffic from data centers and clients that do not accept cookies can catch an AI crawler as a side effect. CMS security plugins can also update user-agent blocklists without notifying the site owner.

Cloudflare’s one-click “AI Scrapers and Crawlers” rule, the “AIndependence” toggle launched in July 2025, groups commercial uses together in blocking a broad list of verified AI crawlers. AWS WAF managed rule groups, Akamai Bot Manager’s AI category, Imperva, and Vercel’s optional “AI bot challenge” are other controls to examine. The rule may also live closer to the content. A shared hosting setup can put nginx in front of Apache while the block sits in.htaccess, leaving the content team with control over only part of the path.

Cloudflare introduced separate Search, Agent, and Training categories on July 1, 2026, then applied updated defaults to new domains starting September 15, 2026. On pages displaying ads, those defaults block Training and Agent crawlers while allowing Search crawlers. For new domains onboarding after that date, a pre-launch review needs to cover all three categories individually. This is not a reason to discard security controls, but to review their marketing consequences deliberately.

The three classes of AI crawler and why collapsing them into one block is the most expensive mistake a publisher can make

What does an “AI bot” actually want from a page? Training crawlers collect content to update a model’s parametric weights during training runs. Blocking them can gradually weaken how well models represent a brand and its category in their stored knowledge. That is a long-term trade-off separate from the decision to permit live retrieval. A defensible policy for many publishers is to block training crawlers while allowing retrieval and search.

Retrieval crawlers build the indexes AI search products query at answer time. OAI-SearchBot serves OpenAI/ChatGPT search, Claude-SearchBot serves Anthropic/Claude web answers, and PerplexityBot serves Perplexity’s search index. Blocking these crawlers cuts off the access they need for direct citation. A robots.txt audit of 47 SaaS and developer-tool domains found OAI-SearchBot named in half as many files as GPTBot, and Claude-SearchBot in roughly a quarter as many as ClaudeBot. Those gaps suggest that policies often name the training agent but leave out the citation agent.

ChatGPT-User, Claude-User, and Perplexity-User fetch pages because someone has just asked for them inside an assistant. Blocking them can give an interested prospect an error when they request content. Separate user-agent strings make these decisions independently controllable, so GPTBot is not OAI-SearchBot, ClaudeBot is not Claude-SearchBot, and Amazonbot is not Amzn-SearchBot. A blanket rule erases those distinctions.

Google-Extended is an important exception to that naming pattern. It is a robots.txt control token, with the search engine’s crawler performing the fetching. Treating it as a separate firewall target without isolating the traffic also blocks the search engine’s crawler and therefore Google Search. A policy meant to limit AI use can thus create a search-access problem.

How to find which layer is actually blocking AI crawlers on your site before changing anything

To diagnose the refusal, trace the request before editing permissions. Up to four places can reject it, namely the CDN or WAF, a reverse proxy, the web server, and the application. Start with the same page and use curl to send a browser user agent, then GPTBot, ClaudeBot, and PerplexityBot. The command `curl -i -A 'GPTBot' ' shows the response headers and body when the address and agent are changed for each test. Compare the HTTP status, Server header, and content that comes back.

A Server header naming cloudflare and a body saying “Sorry, you have been blocked” points to a WAF custom rule, managed rule, or AI bot toggle. A cloudflare response with the cf-mitigated header set to challenge indicates a challenge that crawlers cannot solve. It functions as a block. If the Server header names nginx, Apache, or LiteSpeed and the response has a small plain error page, the block likely comes from the web server or something behind it.

Do not stop at the status code alone. An HTTP 200 response containing a CMS login page or an error can indicate an application-layer plugin blocking access to the requested content. Send the requests again to /robots.txt with the same AI user agents. If that file is also refused, the block runs before its permissions can guide the crawler.

Then compare those results with a made-up agent such as ExampleFetchBot/1.0. If that agent receives 200 while AI agents receive 403, the rules are targeting AI crawlers specifically. That distinction narrows the search for the responsible setting. That alone does not show that the site owner chose the policy on purpose.

The configuration changes that actually open access for retrieval and user-triggered crawlers

Once the blocking layer is identified, align every layer with a selective policy. In robots.txt, the training controls can block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, and Bytespider. Retrieval and user-triggered permissions should allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, and Perplexity-User. The declaration still has a necessary role. On WordPress, check whether an SEO plugin generates a virtual robots.txt, because a physical file can conflict with it and require changes through the plugin editor instead.

At Cloudflare, review “AI Scrapers and Crawlers” under Security, then Bots. The toggle blocks retrieval crawlers as well as training crawlers, so it cannot take the place of rules that treat them separately. Under Security, then Security rules, custom rules can use the Block action for the user agents of training crawlers while leaving retrieval and user-triggered agents out. Use AI Crawl Control before and after the change to review bot requests and robots.txt compliance.

Do not transfer the Google-Extended token into that WAF policy as though it identified a separate crawler. Blocking its fetching traffic also blocks Google's search crawler and Google Search. For new domains onboarding after September 15, 2026, also audit Search, Agent, and Training explicitly before launch. In particular, allowing Search does not establish that Agent traffic can pass on ad-displaying pages.

The origin needs its own review after the CDN. Rules in nginx configuration, Apache configuration, or.htaccess may still refuse a request that passes the edge. Audit CMS plugin settings and update logs as well, since their blocklists can change silently. None of these permissions can be replaced with llms.txt. That file is a complementary Markdown navigation guide for LLMs at inference time.

Why getting crawled is a prerequisite but not sufficient for AI citation

A successful fetch opens a second question. Can the system extract a useful answer from the page, and does that answer deserve a citation? Malformed HTML, too much navigation, and content rendered through JavaScript that the crawler does not run can keep it from extracting an answer from the page.

Even readable content can lose at the generation stage. The model may not have a clear grasp of the brand, the page may miss what the question asks, or another page may answer it more directly. Speed can also affect extraction, but it should not be treated as a simple pass-or-fail gate. The reported relationship between First Contentful Paint and ChatGPT citation is a gradient correlation.

Quantitative, verifiable claims give an answer engine a specific statement it can support. Sections also need enough prose between headings to supply a coherent answer. External context contributes another signal. Brand and entity mentions in authoritative sources correlate more strongly with citation than backlink count alone, so on-page structure is not the whole explanation for visibility.

Measuring whether the fix worked — verifying AI crawler access and citation outcomes

To make the change accountable, measure access and citation separately. Re-run the curl probe against the same pages after each configuration change, this time checking OAI-SearchBot, Claude-SearchBot, and PerplexityBot explicitly. Confirm that each receives 200 for both the page and /robots.txt, and inspect the body for the actual content. Cloudflare’s AI Crawl Control and Bot Analytics can then show the traffic that comes from verified AI crawlers. Use that traffic to establish whether retrieval crawlers are reaching the site at the expected cadence.

Citation verification asks a different question from Google ranking. Query AI answer engines with the prompts the audience would ask, then check whether the brand, content, or URLs appear in the response. Compare the results before and after the infrastructure change. Test Perplexity, ChatGPT, and Claude separately because they have distinct crawlers, indexes, and citation rates. Low overlap across platforms means success in one engine does not establish visibility in the others.

Each new page needs a post-publish access check and a citation check for queries it should answer. That creates a useful boundary between two kinds of work. A refusal sends the team back to infrastructure, while an accessible but uncited page calls for scrutiny of extraction, relevance, and entity presence. CDN configuration then becomes a measured part of AI visibility rather than an assumption hidden beneath the content strategy.

More in Features