Why is my page not appearing in the search engine index?

A page may not appear in the search engine index because it is blocked from crawling, marked as noindex, inaccessible to search engine bots, treated as a duplicate, or has not yet been discovered and processed. Check the page’s technical directives, HTTP response, canonical URL, internal links and sitemap before requesting a new crawl.

Start by confirming whether the page is actually indexed. Search for the exact URL using the search engine’s URL inspection tools rather than relying only on a normal site search. A site search can be incomplete, while URL inspection usually indicates whether the page was crawled, whether indexing is allowed and which canonical URL was selected. If the inspected URL is not indexed, review the reason given and test the live page after making any changes.

The most common causes are set out below.

• A noindex directive is present. Check the page source and response headers for a meta robots directive or an X-Robots-Tag containing noindex. This can be added by an SEO plugin, content management setting, staging configuration or template rule. Remove it when the page should be included in search results, then save the page and confirm that the live version no longer returns the directive.
• Robots.txt is preventing crawling. A disallow rule in robots.txt can stop crawlers from fetching the page or its supporting resources. Robots.txt does not directly remove an already indexed URL, but it can prevent search engines from seeing updated content or confirming other directives. Check the specific path, as well as broader rules affecting its directory.
• The page returns an unsuitable HTTP status. An indexable page should normally return a successful response. Redirects, not-found responses, server errors, authentication requirements and intermittent availability can prevent indexing. Check the URL without being logged in and test both the preferred protocol and hostname versions. Resolve redirect chains and server reliability issues before submitting the URL again.
• The canonical URL points elsewhere. A canonical tag tells search engines which version of similar pages should be indexed. If the page canonicals to another URL, has conflicting canonical signals or uses an incorrect protocol, hostname or path, the other URL may be selected instead. Use one consistent canonical URL and ensure it is accessible, indexable and internally linked.
• The page has not been discovered or crawled yet. New, recently amended or poorly linked pages can take time to find and process. Add a relevant link from an already accessible page, include the URL in the XML sitemap and ensure the sitemap is current. A crawl request can help prioritise discovery, but it does not guarantee immediate indexing or ranking.
• The page is difficult to access or render. Excessive loading times, blocked scripts or stylesheets, broken resources, bot protection, unstable hosting and content that appears only after complex client-side rendering can all affect processing. Make the main content available in the delivered page where practical, reduce technical errors and check that important resources are not blocked.
• The content is substantially duplicated or provides limited additional value. Search engines may choose one version of near-identical pages, parameter URLs, print versions or thin location and service pages. Consolidate overlapping pages where appropriate, improve genuinely useful content and use canonicalisation or redirects for redundant versions. Adding text alone will not resolve duplication if the underlying pages serve the same purpose.
• The URL is in a different property or environment. Check for protocol changes, alternate hostnames, subdomains, development copies and old URL structures. A page may be indexed under a different version of its address, or the live page may still contain settings carried over from a staging site.

Use a structured diagnostic process. First, open the live URL in a private browser window and confirm that it loads without authentication. Then inspect the HTTP status, robots.txt rules, meta robots directives, canonical tag and rendered content. Review internal links and the XML sitemap, and inspect server logs if available to establish whether search engine crawlers are reaching the page. Finally, use the relevant search engine inspection tool to request crawling after all blocking issues have been fixed.

Do not repeatedly request indexing while a page is still blocked or returning errors. The request only asks a search engine to recrawl the URL; it does not override a noindex directive, robots.txt rule, canonical selection or quality decision. After making changes, allow time for the page to be recrawled and check the inspection result again.

If the technical checks are clear but the page remains excluded, compare it with similar indexed pages on the site. Look for weak internal linking, overlapping intent, automatically generated variations, missing primary content or a page that does not provide a distinct reason to exist. Strengthen the page where necessary, link to it from relevant sections and keep its canonical, sitemap and indexing signals consistent.