Why a page is not indexed: debug the earliest failed state

Write the symptom as a URL observation

Begin with the exact preferred URL, the report or search observation, its timestamp, and why the page should own a distinct answer. ‘Not on Google’ is too broad: the URL may be undiscovered, unfetched, blocked, redirected, canonicalized elsewhere, technically eligible but unselected, or indexed yet not served for the tested query. Capture the response chain, rendered directives and content, sitemap state, internal links, canonical variants, and any inspection result. The artifact is an earliest-failure tree. It orders branches by dependency so a late-stage label does not distract from a broken response. Google’s crawling overview and status documentation support this staged diagnosis, while neither offers a guarantee that a corrected URL will be indexed.

Evidence: Google Search Central; Google Search Central

Preserve deployment history before testing causes

Preserve the production state and recent deployment history before attempting a fix. Repeatedly changing the page can reset evidence and make the cause harder to distinguish.

Evidence: Google Search Central; Google Search Central

Branch A: no reliable discovery route

Check whether maintained pages expose crawlable anchors to the preferred URL, whether the sitemap contains the exact final URL, and whether old variants dominate internal references. This branch gains support when the page is valuable and technically healthy but absent from the site’s discoverable inventory. It weakens when current crawl evidence or inspection shows that the URL is already known. Add a justified internal route and repair sitemap accuracy; do not create sitewide links solely to raise a score. A sitemap is a discovery hint, not an indexing directive. If the page has no distinct reader job, an ‘orphan’ warning should trigger an editorial disposition rather than link insertion.

The contradiction test is simple: verified discovery evidence moves the tree forward, while continued absence from a platform report alone does not prove discovery still fails.

Evidence: Google Search Central; MDN Web Docs

Branch B: fetch or rendering does not produce the intended page

Follow status codes and redirects to the final response. Inspect server errors, loops, soft-error content, blocked resources, empty client-rendered shells, and mobile or consent states that replace the main answer. This branch is supported when a crawler or reproducible request cannot receive a stable successful response with meaningful content. It weakens when raw and rendered inspections consistently show the intended page. Repair operations or rendering before editing metadata. A 200 code on a custom ‘not found’ template is not healthy; a visible browser page may still depend on interactions a crawler cannot use. Google’s status-code guidance grounds transport interpretation, while the exact incident requires production evidence.

Retest the same URL and environment after one repair. If transport becomes stable, continue to directives rather than declaring the entire indexing incident solved.

Evidence: Google Search Central; MDN Web Docs

Branch C: publisher controls exclude or consolidate the URL

Inspect robots.txt access, page and header noindex directives, authentication, and the canonical cluster. Then compare declared canonical, redirects, internal links, and sitemap entries. This branch gains support when the publisher’s own controls intentionally or accidentally tell search systems not to retain this URL as representative. It weakens when access is allowed, noindex is absent, and canonical signals consistently prefer the tested URL. Correct the control at its owner rather than layering contradictory hints—for example, do not keep an obsolete URL in the sitemap while redirecting it elsewhere. Google’s canonical documentation explains that declarations are signals and a different representative can still be selected.

The tree should distinguish ‘publisher prefers another URL’ from ‘search system selected another URL despite coherent preference,’ because the available actions differ.

Evidence: Google Search Central; Google Search Central

Branch D: eligible content is not selected

When discovery, transport, rendering, directives, and canonical intent are coherent, investigate whether the page contributes a distinct and useful answer. Compare it with the selected canonical and neighboring pages at the level of promise, evidence, and completion state. Thin variants, near-duplicates, expired information, or pages created primarily for search demand may remain unselected even without a technical block. This is a hypothesis, not a platform mind-reading exercise. Strengthen, merge, or retire content based on reader value and ownership, not by adding boilerplate length. The branch is weakened when a specific earlier technical contradiction is found. It can remain unresolved when evidence does not show why the system has not selected the page.

Do not call eligibility a failure or keep requesting recrawls without a material change. An honest unresolved state is more useful than speculative fixes across the site.

Evidence: Google Search Central; Google Search Central; MDN Web Docs

Repair the earliest supported branch and wait appropriately

Choose one branch with reproducible evidence, write what would disprove it, and make the smallest reversible correction. The next action is to complete an earliest-failure row for one URL and assign its technical or editorial owner. After release, validate production state immediately, then allow an observation window without repeated no-op submissions. Limits are explicit: reports and caches lag, crawl timing is not controlled by the publisher, canonical selection can differ, and no technical change guarantees inclusion or traffic. If two branches remain plausible, gather the cheaper discriminating evidence rather than fixing both. Close the incident when publisher-controlled states are coherent and the remaining selection uncertainty is recorded. Reopen on a new contradictory observation, not merely because a preferred result has not appeared on demand.