Computational Marketing
FeaturesLong read

Fake AI crawlers are probing your site for secrets: what your bot rules and agent-analytics are getting wrong

Spoofed AI crawler names are probing for credentials, so verify IPs.

· 8 min read
Cover illustration for “Fake AI crawlers are probing your site for secrets: what your bot rules and agent-analytics are getting wrong”
Features · November 18, 2026 · 8 min read · 1,822 words

Fake AI crawlers expose a shared weakness in site security and AI-visibility measurement. A user-agent name proves nothing about who sent a request, yet access rules and analytics often trust it. The correction is the same for both systems. Verify crawler addresses against the right published IP ranges before granting identity-based access or counting visits as evidence of genuine AI-crawler reach.

Why UA strings became a trust layer nobody designed them to be

A user-agent string says who a visitor claims to be. HTTP has no verification mechanism baked into that declaration. When AI labs published names such as ClaudeBot, GPTBot, and PerplexityBot, they gave site owners a way to spot the traffic they expected. But recognition is not authentication. Anyone can type the same text into a request, so a rule that trusts the name alone trusts a convention.

A firewall may allow the request, a security tool may lower its priority, and an analytics dashboard may count it as an AI-bot visit. All three inherit the same unsupported claim. Robots.txt rules also depend on declared crawler identities, but publishing instructions for a named crawler does not establish that a visitor using its name is genuine.

What attackers did once they understood that gap

Cequence Security tracked a threat actor impersonating roughly 32 AI bots from late September through late October 2026. The identities included crawlers and assistants associated with OpenAI, Anthropic, Perplexity, Meta, and others. These were not simply visits to public articles. The campaign requested roughly 90 file paths associated with secrets, including .env configuration files, cloud credentials at paths such as /.aws/credentials, private SSH keys, and server environment data.

An environment file is an especially valuable target because it usually holds database passwords, cloud access keys, and API tokens. If publicly reachable, it can expose those secrets through a single request. Cequence also found requests for configuration files written by AI coding assistants, including /.claude/settings.json, /.codex/config.toml, and /.aider.conf.yml. The target set had expanded with developers’ tools. Scanners were now looking on public sites for files made for local development.

GreyNoise’s findings require an important distinction between attempted access and successful theft. Its research documented requests, with extraction of the files being sought remaining unconfirmed. That limits what can be said about damage, but not what can be said about intent or exposure. Across the findings, familiar AI names served as cover for probes aimed at credentials and configuration files, rather than the public content those names would ordinarily suggest.

The Google-Extended edge case that proves the point exactly

Google-Extended makes it unusually easy to establish whether a claimed identity is genuine. Google defines it as a robots.txt token publishers use to opt out of AI training. GreyNoise’s report quotes Google’s documentation saying it “doesn't have a separate HTTP request user agent string.” There is no legitimate sender to confuse with an impostor.

GreyNoise reported 263,849 sessions that used Google-Extended as a user-agent string in 2026. Every one was therefore forged by definition. A rule trusting a recognized AI-related name would still have accepted that claim, even though the name could not identify a genuine crawler request. What would such a rule actually verify? Only that the sender knew which text looked familiar enough to suggest a relationship with Google.

Diagram: Google-Extended: 263,849 Sessions, Zero Legitimate Ones. Visualizes: Visualize a single stark contrast: Google-Extended is documented by Google itself as a robots.txt-only token with no HTTP user-agent string — meaning any request using it…

How the behavior of real crawlers exposes the fakes

GreyNoise found that none of the six forged crawler identities requested /robots.txt during the time it watched them. Its 2026 analysis measured genuine ClaudeBot traffic over the same window and by the same method, with sharply different results. For ClaudeBot, /robots.txt accounted for 12% of requests and was the most-requested path. It never requested a credential file.

That difference follows from what a legitimate crawler is meant to do. It retrieves public web content for indexing or citation, which gives it no reason to request environment files, cloud credential stores, or private keys. Timing still needs care. Real crawlers cache robots.txt rather than fetching it on every visit, so a single request without a preceding robots.txt fetch proves little. Absence across multiple days is the suspect signal.

Origin addresses provide the second part of the diagnosis. GreyNoise compared all 824 source addresses with the published ranges for OpenAI, Anthropic, Google, Perplexity, and Amazon, and none matched. Wrong paths and wrong origins together give operators near-certain separation without waiting for cryptographic identity. These checks work with existing tooling. Yet most dashboards omit the behavioral view that would distinguish a crawler reaching public content from a scanner reaching for secrets.

Diagram: Real Crawler vs. Fake: What the Paths Reveal. Visualizes: Show a behavioral contrast between genuine ClaudeBot traffic and the six forged crawler identities GreyNoise monitored over the same window.

Why name-based rules also corrupt your AI-visibility measurement

The same impostor can slip past security and make the analytics look reassuring. A dashboard filtering “ClaudeBot visits” by user-agent text cannot separate genuine Anthropic requests from scans wearing that name. Its counting method has no authentication step.

For teams pursuing GEO/AEO, the goal is to make content eligible to be cited in AI-generated answers. They need to know whether legitimate crawlers reach the pages meant for them, which takes more than seeing familiar names in traffic logs. Per-path visibility supplies that missing context. Without it, a rise in supposed AI visits can be mistaken for improved content reach when the name-matching layer cannot establish who visited or what they sought.

What most sites are actually doing — and how wide the gap is

DataDome tested hundreds of thousands of live websites with a spoofed assistant user agent and found that the vast majority neither blocked nor challenged the impersonation. Fastly recorded millions of requests impersonating AI crawlers and fetchers within a single day. The exposure is not a niche experiment. One finding shows how broadly sites accept the disguise, while the other shows the volume at which that disguise is used.

Known Agents counts a visit as spoofed when it claims a recognized agent identity but fails that agent’s own authentication. Its data identifies Googlebot as the most impersonated identity, with ChatGPT-User, OAI-SearchBot, ClaudeBot, GPTBot, and PerplexityBot each significant. The problem spans both established search crawlers and newer AI agents. It is not confined to one vendor’s naming scheme or one type of content retrieval.

Known Agents also reports very high robots.txt compliance among bots, which can sound reassuring until the population is considered. That measure describes bots identifying themselves honestly. A scanner using another agent’s identity falls outside that group, so the fact that honest bots follow the rules says nothing about safety from impersonation. The findings show that probing is widespread and ongoing. The practical implication is that most sites have likely already encountered it without recognizing what their crawler labels concealed.

The fix: IP-range verification as the interim standard

For teams correcting this gap, the baseline is to verify the source address before trusting a crawler identity. OpenAI publishes separate lists for GPTBot, ChatGPT-User, and OAI-SearchBot, and each name requires comparison with its own list. A single “OpenAI” allowlist is not an adequate substitute.

Each vendor list should have someone responsible for it and a schedule for fetching updates. Stale ranges can cause genuine crawlers to be treated as strangers and undermine the trust placed in an allowlist. Maintenance is part of verification. The same verified classification should underpin crawler analytics, so the access decision and the visit count do not rely on different standards of evidence.

Alerts based on what a request does add protection when it claims a trusted name. GreyNoise advises alerting on requests for /.env, /.aws/credentials, and /.git/config regardless of user agent. It also advises treating sessions labeled as crawlers as suspect if they go multiple days without ever requesting /robots.txt. These checks answer a different question from the IP comparison. They establish whether the requested paths fit public-content retrieval, rather than letting a familiar label suppress scrutiny.

Direct protection of the files remains essential. Environment files, .git directories, cloud credential files, and private keys should always be kept out of the web root. Any cloud key that was reachable through a public URL should be rotated immediately and treated as compromised. Crawler authentication cannot make an exposed secret safe. Its role is to make identity-based permissions and measurements defensible, while removing secrets from public paths reduces what any visitor can obtain.

Web Bot Auth and what cryptographic crawler identity will eventually solve

Web Bot Auth addresses the structural weakness by letting bots prove identity cryptographically. It uses HTTP Message Signatures under RFC 9421, an Ed25519 key for each agent, a Signature-Agent header, and a JWKS directory. Cloudflare, Amazon, Akamai, and OpenAI back the effort, and an IETF working group was chartered in 2025. This changes the basis of trust. The visitor supplies a verifiable signature.

Cloudflare has treated signed agents as verified bots since July 1, 2026, labeling them Direct or Intermediary. From September 15, 2026, newly onboarded domains block Training and Agent traffic on pages displaying ads while allowing Search. As a result, unsigned, unverified agents are blocked by default on a growing share of the web that runs ads. Adoption has practical consequences for access. Cloudflare’s registry, in the November 4, 2026 snapshot described in the notes, listed hundreds of directories, with a majority returning valid key sets and most carrying verifiable signatures.

Compatibility still limits what operators can assume. Cloudflare’s verifier follows the pre-October-2025 Signature-Agent structured-string format and rejects the newer dictionary form mandated by the -01 draft. An implementation following that newer specification can therefore fail verification there. Google also leaves a significant gap. As of May 2026, its main Googlebot indexing crawler was not signing requests, while its AI-browsing Google-Agent was. Cryptographic identity is the right basis for trust in the long term, but too few agents use it and formats still differ, so it cannot yet replace IP-range verification.

What this means for teams trying to earn AI citations, not just block bad crawlers

For content teams, verified crawler access is both a security concern and a prerequisite for interpreting AI visibility. IP-confirmed analytics and per-path visibility show which legitimate crawlers reached which content. That is the input evidence. Mentions across ChatGPT, Claude, Gemini, and Perplexity are the output metric, and crawler visits alone do not establish that a page earned a citation.

The content a crawler can retrieve shapes what an answer engine can cite, so publishing and measurement need a trustworthy connection. Real, retrievable content published on a cadence, with structured pages crawlers can parse cleanly, supports that goal. Site maintenance also has a security dimension. Keeping credentials out of public paths protects the same web root that must remain accessible to legitimate crawlers, without confusing openness to public content with access to private files.

Decisions about welcoming or blocking AI bots therefore depend on which visitors can substantiate their claimed identity and which pages they actually reach. Published IP ranges let teams check who a crawler is today, while path-level evidence shows whether genuine crawlers reach the content a team hopes will earn citations. Together, they turn a log of claims into evidence a content program can use.

More in Features