When "AI content optimization" means auditing the markdown copy and llms.txt: a page assistants can read is a test you can run, not a service you book
Check what AI crawlers actually receive from your pages.

AI content optimization begins with a test of what an assistant can actually read. A page’s markdown-like text, crawler permissions, and llms.txt can be inspected directly. That audit exposes missing evidence and blocked access, but it cannot promise recommendations. The useful distinction is between making your site readable and giving an AI engine reason to cite it.
Why AI assistants often read a different version of your site than humans do
A founder opens the company’s site and sees customer logos, a testimonial carousel, and case-study numbers. The crawler log shows a visit to that same page, yet the response body contains little of that evidence. The page looks persuasive in a browser. What did the crawler actually get?
For a client-side React or Next.js single-page application, a crawler that does not run client-side scripts may get an empty shell. Proof points held in client-side components do not appear in that response. Server-side rendering or static generation can instead put substantive content into the returned HTML. Without the visual design, that content reads like a markdown-like document, though the response need not be in Markdown.
That difference changes the evidence available when an assistant composes an answer about your category. A testimonial visible to a visitor may be absent from the text available to the machine. Browser appearance is therefore an incomplete test. Comparing the returned content with the rendered page makes the gap inspectable.
What the three machine-readable files actually control, and which one founders routinely ignore
Three root-level files have different jobs, but the one with the most at stake is also the easiest to miss when discussing AI-specific formats. Robots.txt governs crawler access to paths on a site. It was proposed in 1994 and became a formal standard, RFC 9309, in September 2022. Most major AI crawlers, including GPTBot and ClaudeBot, honor it. An accidental blocking rule can cut off access before the quality of your content enters the picture.
Not every agent follows the same rules. PerplexityBot officially commits to honoring robots.txt, while Perplexity-User, its user-fetch agent, does not, and undeclared Perplexity crawlers have been documented bypassing it. Those exceptions limit what the file can guarantee while leaving the need to inspect it intact. Sitemap.xml has a separate discovery role, listing indexable URLs without providing a curated order of importance.
Jeremy Howard of Answer.AI proposed llms.txt in September 2024 as a plain Markdown map of a site’s most useful pages. Its structure includes an H1 site name, a blockquote summary, and H2 sections containing annotated links. The file cannot grant access, deny access, block training, or enforce anything, so treating it as content protection confuses navigation with control.
Google has publicly said that it does not use llms.txt, so of these two files, robots.txt controls the crawler access that matters for being seen on Google. The value of llms.txt is bounded and dependent on adoption. Its companion, llms-full.txt, combines full site content into one Markdown document for deeper ingestion by agents and coding tools. Most marketing sites need only the shorter file, while developer products with a lot of documentation can benefit from both.
What the crawl logs actually show about which bots read llms.txt
A 12-week crawl study logged AI crawler requests to llms.txt and robots.txt on the same domains from April 27 to July 19, 2026. It covered GPTBot, ClaudeBot, PerplexityBot, and Meta’s crawler. PerplexityBot made no llms.txt fetches across the study domains during that window. The proposed navigation aid was not part of that crawler’s observed activity.
Meta’s crawler was the important exception to the pattern. It fetched llms.txt more often than robots.txt, unlike the other crawlers in the study. That outlier rules out a sweeping claim that bots simply ignore the file. It also shows why a finding about one crawler should not become a claim about all assistants.
What share of a crawler’s total visits includes the file? The number of fetches can look good even when they make up very little of what the crawler does. The study points to llms.txt as a low-probability channel for OpenAI, Anthropic, and Perplexity. Shipping it takes roughly an hour, however, and requests to it can be monitored as a live hypothesis about changing crawler behavior.
That distinction is especially useful when reading claims about vendor support. Much of the circulating llms.txt commentary is AI-generated, and some includes fabricated vendor confirmations. A confident explanation is not evidence of a fetch. What the logs show is a firmer basis for deciding how much work to put into the file.
How to audit the markdown your site actually serves to AI crawlers
The practical response is a short audit that a founder, growth engineer, or technical marketer can work through on a laptop. Start by fetching a priority page with curl, wget, or a headless browser with client-side scripting disabled. Compare the response body with the browser-rendered page. Look for evidence from customers, testimonials, and proof-point numbers. If they appear only after client-side scripts run, the browser is showing evidence that the fetch without script execution does not contain.
Next, open /robots.txt and check the access rules. Check crawler access for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended to ensure they are not inadvertently blocked. A disallow rule can defeat the work of making a page readable, so it belongs before polishing the machine-readable summary.
Then inspect /llms.txt, if it exists. A valid first version needs one H1 for the site name, a blockquote brand summary, and H2 sections grouping annotated links by topic. Use absolute URLs with the full domain because relative paths can break when the file is fetched out of context. Keep the file concise, leaving full-content ingestion to llms-full.txt. A junior developer can ship this first version in under an hour.
The summary deserves particular care because it gives an assistant a ready-made description of the brand. Write one or two sentences that can stand alone. That is an opportunity for accurate reuse. The descriptions on links should likewise say what is on the linked page.
After that, filter server access logs for /llms.txt and /llms-full.txt requests using known AI crawler user-agent strings. For priority pages visited by GPTBot, compare the substantive content expected on the page with the body the site returns. A logged visit and a near-empty response expose a different problem from a file that receives no visits.
Finally, check the readable copy on each priority page. Does it answer a real question in its first two sentences? Does it include at least one sourced statistic or named fact, and do FAQ questions sound like something a person would ask aloud? A fetch that works does not on its own show that the text is worth extracting.
Why the copy decisions that make content extractable are different from the ones that made it rank
An assistant composing an answer needs a passage it can use with confidence. A page can rank well and still make that passage difficult to find. Classic SEO writing often delays its conclusion through a long preamble intended to hold attention. Under context-window constraints, that preamble becomes material to skip.
Consider a comparison page that takes six paragraphs to introduce the category before saying how the options differ. The answer exists, but it is buried. A citation-first version puts the direct comparison in the opening sentences and follows it with supporting detail. The change is not simply shorter writing. It gives the passage a clear purpose even when separated from the rest of the page.
Specific evidence makes that passage more useful. Attributed quotations, inline numeric facts, and named sources provide credibility signals that keyword density does not. The Princeton and Georgia Tech GEO paper found up to a 40% lift in source visibility for content using citations, direct quotes, and statistics. That finding supports an editorial check, with the lift for any particular page still uncertain. Build a five-minute extractability pass into publishing to handle fixes short of a full rewrite.
Content format also changes what a citation can accomplish. Category listicles get a lot of citations, while case studies with names lead to more conversions when buyers are weighing their options. Comparison content sits between those roles. Analyst-quoted content borrows earned authority and tends to achieve a high citation rate per hour of production. Assess both whether a draft is extractable and which buyer question its evidence answers.
How to measure whether AI engines are actually naming and citing your brand
A mention names your brand in the answer. A citation links to your domain as evidence. They can occur separately, so recording only one loses part of the picture. Neither on its own proves that the answer gets the facts about your company right.
The self-service baseline is a fixed set of category and comparison prompts run weekly across ChatGPT, Perplexity, Claude, and Gemini. For each prompt and engine, record mention rate, URL citation rate, sentiment, prominence, and share of voice. A small prompt set can take 30 minutes. A larger one can require several hours, which is the natural point at which automation becomes useful.
Engine differences also affect interpretation of the results. Perplexity retrieves live web results and typically supplies clickable citations, making traffic attribution in GA4 more direct. Gemini’s visibility is more tightly coupled to Google’s live index, so search performance is a major driver there. Low citation overlap across engines means success on one cannot stand in for the others. A complete picture requires running the same prompt set across all four.
Microsoft launched the AI Performance report in Bing Webmaster Tools in public preview in February 2026. In June 2026, it added Intents, Topics, Citation Share, and Compare. The report shows citations, cited pages, grounding queries, page-level activity, and visibility trends across Microsoft Copilot, AI-generated Bing summaries, and select partner integrations. It provides a first-party measurement option for those surfaces. It does not replace observation of the other engines.
Even with that coverage, citation counts remain an imperfect measure of influence. A page can be listed beneath an answer that uses almost none of its information. Its facts may also shape an answer without a citation that can be seen. Counts establish one observable part of the result. They can help show what changed after an audit, but on their own cannot prove that a given edit led to a recommendation.
Where the markdown audit stops and the harder off-site problem begins
A readable site can still be absent from recommendations. AI engines draw on live-retrieved content and on pre-training data, which shapes what a model knows before retrieval begins. Perplexity’s retrieval layer and Bing AI illustrate the live-content side. A brand that rarely appears in pre-training data may have a site that works well technically and still be left out of ChatGPT’s unprompted category recommendations.
Earned media, cited research, named-author bylines, and third-party references place a brand in content it did not publish itself. Those signals can become part of both pre-training data and retrieval-based answers. An llms.txt summary cannot create that presence in other people’s content, however carefully it describes the business.
The audit is therefore the right first move because it removes inspectable blockers at little cost. The next task is different. Building third-party presence requires a publishing and PR strategy over time. A page assistants can read is a test you can run. Becoming a brand they have reason to recommend is the longer work.


