Robots.txt Validator

Paste your robots.txt content to validate directives and catch common mistakes, or fetch /robots.txt from a site.

We fetch /robots.txt from the site to avoid CORS issues.

Robots.txt Validator - Check Crawler Rules Before You Publish

A robots.txt file controls how search engine crawlers (Googlebot, Bingbot, and others) access your site. A small syntax mistake can lead to rules being ignored, or worse, you can accidentally block your entire website from being crawled. This validator helps you quickly spot issues, understand what each group means, and confirm your directives follow the expected format.

You can paste your file directly into the editor or load /robots.txt from a website. The results show errors, warnings, and a parsed view of your user-agent groups so you can review your rules at a glance.

What This Tool Checks

  • Grouping: directives that appear before a User-agent group are flagged.
  • Directive format: lines missing the Directive: value structure are reported.
  • Paths: Allow and Disallow values that don’t start with / are highlighted.
  • Sitemap: validates that Sitemap: values look like valid URLs.
  • Common SEO risks: warns when User-agent: * blocks all crawling via Disallow: /.

Common Robots.txt Mistakes

  • Blocking everything: User-agent: * + Disallow: / on a production site.
  • Missing user-agent: adding Disallow rules without specifying which crawler they apply to.
  • Bad sitemap line: typos, missing protocol, or extra spaces in the sitemap URL.
  • Incorrect paths: forgetting the leading slash (for example, using admin/ instead of /admin/).
  • Confusing indexing with crawling: robots rules control crawling, not guaranteed deindexing.

How to Use the Robots.txt Validator

  1. Paste your robots.txt into the editor, or load it from a website.
  2. Click Validate to parse groups and highlight issues.
  3. Review the Findings for errors and warnings, then adjust your rules.
  4. Re-check after changes to confirm your final version is clean.

Robots.txt Directives Explained

Directive Purpose Example
User-agent Starts a group and defines which crawler the rules apply to. User-agent: *
Disallow Blocks crawling of a path for the current group. Disallow: /admin/
Allow Allows crawling of a path (often used with a broader disallow). Allow: /public/
Sitemap Points crawlers to your XML sitemap location. Sitemap: https://example.com/sitemap.xml
Crawl-delay Suggests a delay between crawler requests (not supported by all bots). Crawl-delay: 10

Frequently Asked Questions

Does robots.txt prevent a URL from appearing in Google?

No. Robots rules primarily control crawling. A blocked URL can still be indexed if it’s discovered through links. For reliable removal, use proper indexing controls (like noindex where supported, or server-side access rules) and request removal in your webmaster tools if needed.

Where should robots.txt be located?

It should be placed in the root of your domain, for example https://example.com/robots.txt. Each subdomain should have its own robots file if needed.

Should I include my sitemap in robots.txt?

Yes, it’s a common best practice. Adding a Sitemap: line helps crawlers discover your sitemap faster.

Why does the validator warn about Disallow: /?

Disallow: / blocks crawling of the entire site for that user-agent group. It’s useful for staging environments, but it’s risky on a live site unless you’re intentionally blocking crawlers.