Why is my URL not appearing in the search engine index?

A URL may not appear in the search engine index because search crawlers cannot access it, the page is explicitly excluded, another URL has been selected as canonical, the page is returning an unsuitable status, or the search engine has not yet processed it. Use Search Console’s URL Inspection report to identify the specific reason, correct the underlying issue, and then request indexing again.

Before changing anything, confirm that the URL is genuinely missing from the index. A site search can be useful as an initial check, but it is not definitive. Inspect the exact preferred URL, including its protocol, subdomain, capitalisation and trailing slash. Search Console will normally indicate whether the URL is indexed, excluded, or currently unknown to the search engine.

Common causes of non-indexing include:

• A noindex directive: The page may contain a noindex robots meta tag or an equivalent HTTP header. Remove this directive if the page should be indexed. Check the page source and any SEO or publishing settings that may be adding it.
• Crawling blocked by robots.txt: A robots.txt rule can prevent crawlers from fetching the page. Review the relevant rule carefully, particularly if a broad directory-level directive affects the URL. Robots.txt controls crawling; it is not the correct method for reliably removing an already known URL from the index.
• An incorrect canonical URL: If the page declares a different canonical URL, or if other signals consistently identify another version as preferred, the search engine may index the canonical page instead. The canonical should point to the accessible, preferred version, and internal links, redirects and sitemap entries should support the same choice.
• An unsuitable HTTP status: Pages returning a client or server error, repeated redirects, an unexpected redirect destination, or an authentication requirement cannot normally be indexed as intended. The preferred URL should return a successful response and should be accessible without a login, unless it is deliberately restricted.
• A soft 404: A page can return a successful response while appearing empty, unavailable or substantially equivalent to a missing page. Provide meaningful content where the URL is valid, or return a proper not-found or gone response where it is not.
• Duplicate or near-duplicate content: Search engines may choose one version from several substantially similar URLs. Review parameters, print versions, filters, regional variations and duplicate pages, then consolidate signals with redirects or canonicals where appropriate.
• Insufficient discovery signals: A page that is not included in an XML sitemap and has few or no internal links may take longer to discover. Link to important pages from relevant, crawlable areas of the site and include their preferred URLs in the sitemap.
• Content that does not provide enough distinct value: Pages with very little original content, automatically generated text, or content that closely repeats another page may be crawled but not selected for indexing. Improve the page’s usefulness, clarity and distinct purpose rather than adding text solely to increase length.
• Temporary processing or crawl issues: Submission does not guarantee immediate indexing. A newly published or substantially updated page may still be awaiting crawling, rendering, quality evaluation or other processing.

Use the URL Inspection report to work through the issue in a controlled order. First check the indexing status and the stated reason for exclusion. Then review the live URL test to confirm that the current page is reachable and that its robots directives, canonical and rendered content are correct. Where available, inspect the referring page and sitemap information to check whether the URL can be discovered through normal site signals.

After making a correction, allow the change to be deployed consistently across the site. Test the live URL again, confirm that the page returns the expected status and is no longer blocked, and submit a fresh indexing request. A request asks the search engine to recrawl the URL; it does not override a noindex directive, guarantee inclusion, or guarantee a particular ranking. Repeatedly submitting the same unchanged URL is unlikely to resolve a technical or content problem.

If the URL is indexed but does not appear for a particular search, that is a ranking or search-result visibility issue rather than an indexing issue. Check the exact indexed URL, canonical selection and page content before investigating relevance, competition and search intent. If several important URLs are excluded for the same reason, review site-wide templates, robots rules, sitemap generation and SEO settings rather than correcting each page individually.