What the validator checks
The validator reads your file line by line the way RFC 9309 describes a robots.txt parser: anything after # is a comment, field names are matched without regard to case, and consecutive User-agent lines are collected into one group until a rule follows. From that parse it reports errors, warnings and a card for each group, so you can see which rules apply to which crawler. It is a syntax and structure check; it does not tell you what any search engine has indexed.
Paste a file into the editor and click Validate, or type a site name and click Load robots.txt to pull the live file. After a fix, run it again; the parse happens in your browser and takes a moment.
What happens when you load a file from a website
Browsers block a page on one domain from reading robots.txt on another, so the Load button asks our server to fetch it. The server requests https://yourdomain/robots.txt (or http:// if you type it), follows up to five redirects, waits up to eight seconds and reads the first 200 KB. It only connects on ports 80 and 443, only over IPv4, and refuses private and loopback addresses. The text comes back to your browser and is parsed there; we do not keep a copy. If the fetch fails, the message says why: a 404 means the site has no robots.txt, which for Google means crawl everything; a 403 usually means the site blocks unfamiliar clients (ours identifies itself as RobotsTxtValidator/1.0), in which case open the file in your browser and paste it in.
Robots.txt directives explained
| Directive |
Purpose |
Example |
User-agent |
Starts a group and names the crawler token the rules apply to; * means any crawler without a more specific group. |
User-agent: * |
Disallow |
Blocks crawling of paths that start with the value. An empty value is ignored, so it blocks nothing. |
Disallow: /admin/ |
Allow |
Permits crawling of a path, usually to re-open part of a disallowed folder. |
Allow: /public/ |
Sitemap |
Gives the absolute URL of an XML sitemap. Not part of RFC 9309, but read by Google and Bing. |
Sitemap: https://example.com/sitemap.xml |
Crawl-delay |
Asks for a pause, in seconds, between requests. Honoured by Bing, ignored by Google. |
Crawl-delay: 10 |
Need to write a file rather than check one? The Robots.txt Generator builds the groups and the sitemap line, and the Meta Tag Generator writes the noindex tag that actually keeps a page out of results.