Computational Marketing
FeaturesLong read

Technical SEO audits are a diff, not a score: script the gap between the pages buyers need and the pages Google indexed

Audit the missing pages, trace each to its failure stage, and fix by revenue impact.

· 8 min read
Cover illustration for “Technical SEO audits are a diff, not a score: script the gap between the pages buyers need and the pages Google indexed”
Features · November 21, 2026 · 8 min read · 1,835 words

A technical SEO audit should produce a diff. Its central output is the gap between the pages buyers need to find and the pages Google has indexed, with each missing URL tied to a failure stage and a cause. That approach also exposes a second gap, between pages AI systems can access and pages they actually cite.

Why the gap between pages buyers need and pages Google indexed is the right unit of audit

A health-score dashboard shows a set of warnings. A growth engineer needs to know whether the pricing page is in Google’s index. A single score can make many small faults look much like one missing page that does serious harm. A stray noindex on a revenue-generating page belongs in a different class from minor warnings elsewhere.

The useful unit is therefore the missing page. More precisely, among the URLs that should be available to buyers, the missing ones are those Google has not actually indexed. Not every URL a crawler finds belongs in that first set. Redirects, duplicates, and pages grouped on purpose under another canonical URL are not the same loss as a missing product or pricing page.

This changes the order of the audit. Speed, structured data, and other checks cannot compensate for a page that cannot enter the index, because that page cannot rank regardless of how quickly it loads. Those checks still have a place, but index eligibility comes first. The audit moves from “How healthy is the site?” to the more concrete “Which needed pages are missing, and what stopped them?

How the Google index pipeline stages a gap into three distinct failure modes

A missing URL can fail at different points. Google may not have discovered it, may know about it without crawling it, or may have crawled it without including it in the index. These are not interchangeable states. Each puts the next investigation in a different part of the pipeline. Stage attribution is the first step toward a useful fix list.

Google Search Console’s Pages report distinguishes “Discovered — currently not indexed” from “Crawled — currently not indexed.” The first means Google knows the URL exists but has deprioritized crawling it, usually because of crawl-budget or quality signals. The second means Google reached the page but declined to include it, usually because of quality, duplication, or thin content. Why spend time proving that Google can find a URL when the evidence already says it fetched the page?

That said, the report is not a live record of every crawl. Search Console data is sampled and lags actual crawl activity, so it cannot settle every question about what Googlebot did. A sitemap says which pages the site wants crawled. Logs show whether Googlebot reached them, making the distinction between a discovery problem and a later rejection more concrete.

The four causes that explain most of the gap, and the order they appear in a sitemap

Once the stage is known, the audit turns to attribution. Four causes explain much of the gap, namely sitemap hygiene, robots directives, canonical conflicts, and crawl-budget waste. Their order follows the path toward indexation. Did the site tell Google about the page, let it in, point to the right version, and leave enough crawl capacity to reach it?

A sitemap is a statement of intent. When it includes redirected, noindexed, or canonicalized-away URLs, it sends Google toward pages that should not become independent index entries. Those inclusions waste the crawl budget the sitemap is meant to protect. Only canonical, indexable URLs belong there. Otherwise, the intended page set is noisy before the audit even compares it with the index.

Robots directives can then contradict that intent outright. A site-wide disallow left in robots.txt after a staging deployment can keep Googlebot out even when the site looks normal in a browser. Reading the live file line by line belongs at the start of an audit, even though sitemap intent comes first in the causal model. A forgotten noindex creates a different failure. Googlebot can reach the revenue-generating page, yet the instruction excludes it from the index.

Canonical conflicts create ambiguity. Each page should have a self-referencing canonical unless intentional consolidation is the goal, so a missing or mismatched canonical warrants investigation. Crawl-budget waste comes later on the same path. Parameterized duplicates, faceted navigation, soft 404s, and redirect chains consume attention that should reach priority pages, leaving fresh content waiting while seasonal demand can pass. Chains of three or more hops compound that waste and can degrade the PageRank signal passed to the final destination.

Why JavaScript rendering makes the gap wider and harder to see

JavaScript adds a delay that browser checks can conceal. A page can look complete to a person while its initial HTML lacks the headings, body copy, and navigation links that make it useful to a crawler. Googlebot does not execute JavaScript synchronously. Instead, JS-dependent URLs enter a queue for a second rendering pass, which can delay indexing by days.

That distinction makes view-source the first JavaScript check. Browser DevTools can show a page after scripts have changed it, while the audit needs to know what was present before JavaScript ran. Are the primary headings, body copy, and navigation links already in the initial HTML? If not, a crawler that does not execute JavaScript cannot see that content. For Google, the same absence introduces a rendering dependency between the first fetch and possible indexation.

The before-and-after contrast is simple but consequential. Content absent from the initial source must wait for rendering. If that wait lasts days, the index gap stays open even when a check in a browser seems to pass.

The remedy sits in the rendering architecture. Next.js, Nuxt, Remix, and Astro support server-side or static rendering on indexable routes, so the audit finding need not imply a whole-application migration. Using that rendering for SEO templates is the smallest fix that can work here. The audit’s job is to identify which needed pages depend on rendering and whether that dependence is blocking access.

What the diff looks like in practice: running the gap from baseline to attributed list

To turn that diagnosis into an audit output, begin with the baseline. The first comparison is between Search Console’s indexed URL count and the site crawler’s URL count, while the sitemap gives a rough count of the pages the site wants indexed. Counts reveal the scale of the discrepancy. They do not identify the missing pages or explain why a crawled URL should belong in the index. The URL-level comparison supplies that detail.

Next, separate the Pages report into Discovered, Crawled, and Indexed buckets. Each bucket becomes a distinct part of the diff. For discovered URLs that remain uncrawled, attribution starts with sitemap cleanliness and crawl budget. For crawled-but-unindexed pages, it turns to quality signals, duplication, and thin content. Where an indexed page has the wrong canonical, the investigation traces the canonical chain.

Server logs then show whether what the bots did fits those explanations. Orphan pages deserve particular attention because they have no internal links, making them functionally invisible through the site’s navigation regardless of content quality. A known orphan URL may appear in the Discovered bucket with no evidence of crawling in the logs. An orphan Google has not discovered is at an even earlier stage.

The final output is an attributed list ordered by business impact. A missing revenue-generating page ranks above an over-indexed tag archive, even if the archive produces more warnings. The recommended cadence is quarterly for enterprise sites and semiannual for mid-sized organizations, with additional audits after a migration, a traffic drop, or a Google core update, and before a planned investment. CMS updates and plugin changes can reopen a previously closed gap. The list therefore needs a repeatable baseline.

Why AI crawlers extend the gap into a second diff the same audit must produce

Googlebot’s ability to render a page does not establish that an AI crawler can read it, so a Google-focused audit leaves part of the needed-page set untested. Vercel and MERJ’s December 2024 analysis found no JavaScript execution by GPTBot, ClaudeBot, or PerplexityBot. For those crawlers, content missing from the initial HTML is unavailable through that fetch.

The technical questions therefore extend beyond Googlebot’s access. Robots.txt policy sets whether the named crawlers are allowed to reach the content, while rendering shapes what they can read when they get there. A page available after Google’s rendering pass may remain inaccessible to a non-rendering bot. Passing the Google check does not settle the AI check.

Google AI Overviews require inclusion in Google’s index, ChatGPT with Search retrieves from Bing’s index, and Perplexity relies on its own proprietary crawl index. The broader principle is index-specific. A page must be findable through the system an engine uses before it can become a citation candidate.

Machine readability comes after that access layer. Schema.org structured data helps language models pull content from a page, but they still need to be able to reach it. If an older post repeats a figure that a newer page corrects, the grounding system can interpret the conflict as a quality signal and cite neither. The second diff must therefore distinguish inaccessible content from accessible content whose conflicting statements undermine its use.

Measuring the AI citation gap so the diff has a feedback loop

Access establishes eligibility for citation. That is why the second diff needs a measurement loop beyond the crawl audit itself. For Google, the loop already has familiar inputs, including indexed counts over time, crawl stats that expose budget anomalies, and server logs that show bot activity. Together, they reveal whether a fix closed the gap or whether a later change reopened it. AI visibility needs the same kind of record of what happened.

An AI brand tracker supplies that record. It runs a library of buyer prompts across AI engines on a schedule, logs whether and where a brand is cited, and tracks changes over time. In that sense, it is the AI-search equivalent of a keyword rank tracker. The prompt library gives the measurement a defined scope. Without that scope, “the brand is absent from AI” is too broad to serve as a diagnostic finding.

The record must distinguish three different signals. Adobe identifies them as being mentioned in direct responses, being cited as a source through a URL, and being recommended in answers. A mention is any appearance in an answer, while a recommendation explicitly suggests the brand and a citation points to a source URL. These outcomes do not imply one another.

Those distinctions keep the feedback loop honest. An absent citation is not, by itself, proof of a robots block or rendering failure, just as a discovered URL is not proof that Google crawled it. The measured outcome has to be read against the technical evidence before it becomes a fix. More publishing cannot be assumed to resolve an access problem, a canonical conflict, or contradictory source content. The next action should follow the observed gap and its attributed cause.

More in Features