How it Works
This tool performs a comprehensive check of a URL's visibility to search engines. It analyzes multiple layers of crawler control:
- HTTP Status: Verifies if the page is live (200 OK), redirected, or missing (404).
- Robots.txt: Checks if the path is blocked for search engine crawlers in the site's robots.txt file.
- Meta Robots: Scans the HTML for
noindex,nofollow, and other directives. - X-Robots-Tag: Checks HTTP response headers for crawler instructions.
- Canonical: Identifies the preferred version of the page to avoid duplicate content issues.
Indexing Status Explanation
A URL is considered Indexable only if it meets all these criteria:
- Returns a 200 OK status code.
- Is not blocked by robots.txt.
- Does not have a
noindexdirective in Meta tags or HTTP headers.