Robots.txt Validator

Paste your robots.txt content to validate directives and catch common mistakes, or fetch /robots.txt from a site.

We fetch /robots.txt from the site to avoid CORS issues.

What the validator checks

The validator reads your file line by line the way RFC 9309 describes a robots.txt parser: anything after # is a comment, field names are matched without regard to case, and consecutive User-agent lines are collected into one group until a rule follows. From that parse it reports errors, warnings and a card for each group, so you can see which rules apply to which crawler. It is a syntax and structure check; it does not tell you what any search engine has indexed.

Paste a file into the editor and click Validate, or type a site name and click Load robots.txt to pull the live file. After a fix, run it again; the parse happens in your browser and takes a moment.

Findings it reports

  • Rule outside a group (error). An Allow, Disallow or other rule that appears before any User-agent line, or a group with rules but no user agent.
  • Malformed line (warning). A line with no colon, an empty directive name or an empty User-agent value.
  • Path without a slash (warning). Disallow: admin/ instead of /admin/; crawlers match paths from the first character, so the rule would never apply.
  • Bad sitemap or crawl-delay value (warning). A Sitemap: value that is not an absolute URL, or a Crawl-delay that is not a number.
  • Unknown directive (warning). Lines such as Noindex:, which Google does not support in robots.txt.
  • Everything blocked (warning). A User-agent: * group containing Disallow: / with no Allow to soften it.

What it does not check

  • Specific URLs. It does not test whether /blog/post-1 is allowed. When Allow and Disallow overlap, the longest matching rule wins under RFC 9309, and this tool does not run that comparison. Use the URL Checker for a single address.
  • Wildcards. * and $ in paths are accepted but not evaluated.
  • Crawler behaviour. Crawl-delay is checked for format only. Google ignores the directive (How Google interprets the robots.txt specification), while Bing honours it (Bing Webmaster Blog).
  • Indexing. A valid file says nothing about what is in Google's index. Robots rules control crawling; they do not remove a URL from an index.
  • Size. It does not warn when a file passes 500 KiB, the limit Google enforces and the minimum RFC 9309 asks parsers to accept.

What happens when you load a file from a website

Browsers block a page on one domain from reading robots.txt on another, so the Load button asks our server to fetch it. The server requests https://yourdomain/robots.txt (or http:// if you type it), follows up to five redirects, waits up to eight seconds and reads the first 200 KB. It only connects on ports 80 and 443, only over IPv4, and refuses private and loopback addresses. The text comes back to your browser and is parsed there; we do not keep a copy. If the fetch fails, the message says why: a 404 means the site has no robots.txt, which for Google means crawl everything; a 403 usually means the site blocks unfamiliar clients (ours identifies itself as RobotsTxtValidator/1.0), in which case open the file in your browser and paste it in.

How to use the validator

  1. Paste your robots.txt into the editor, or load it from a website.
  2. Click Validate to parse the groups and list the findings.
  3. Fix the errors first, then look at each warning and decide whether it is intended.
  4. Run it again after editing, and once more after uploading the live file.

Robots.txt directives explained

Directive Purpose Example
User-agent Starts a group and names the crawler token the rules apply to; * means any crawler without a more specific group. User-agent: *
Disallow Blocks crawling of paths that start with the value. An empty value is ignored, so it blocks nothing. Disallow: /admin/
Allow Permits crawling of a path, usually to re-open part of a disallowed folder. Allow: /public/
Sitemap Gives the absolute URL of an XML sitemap. Not part of RFC 9309, but read by Google and Bing. Sitemap: https://example.com/sitemap.xml
Crawl-delay Asks for a pause, in seconds, between requests. Honoured by Bing, ignored by Google. Crawl-delay: 10

Need to write a file rather than check one? The Robots.txt Generator builds the groups and the sitemap line, and the Meta Tag Generator writes the noindex tag that actually keeps a page out of results.

Frequently Asked Questions

Does robots.txt prevent a URL from appearing in Google?

No. Google's own documentation calls robots.txt a way to manage crawler traffic, not a mechanism for keeping a page out of Google (introduction to robots.txt). A blocked URL can still be indexed from links elsewhere and shown without a snippet. To remove a page, let it be crawled and add a noindex tag, protect it with a password, or use the removal tool in Search Console.

Where should robots.txt be located?

At the root of the host, for example https://example.com/robots.txt. The file applies only to the protocol, host and port it was fetched from, so https://shop.example.com and http://example.com each need their own.

Should I include my sitemap in robots.txt?

Yes, if you publish one. The Sitemap: line lets any crawler that reads the file find the sitemap without a submission in each search engine's console. Use the absolute URL; the validator flags anything else.

Why does the validator warn about Disallow: /?

Because in a User-agent: * group with no Allow line it blocks every crawler from every path. That is right for a staging site and wrong for almost everything else, so the tool asks you to confirm it is intended.