Robots.txt Generator

Choose which crawlers may fetch which paths, add your sitemap address, and copy or download the finished robots.txt.

Default Rules (All Robots)

This sets the rule for User-agent: *

Specific Crawlers

Restricted Directories

Block crawlers from accessing private folders (e.g., /admin/, /cgi-bin/)

Sitemap Location

Point search engines directly to your XML sitemap.

Your robots.txt File

Save this text as robots.txt and upload it to the root directory of your website.

What a robots.txt file does

A robots.txt file sits at the root of a host, for example https://www.example.com/robots.txt, and tells crawlers which URL paths they may request. Google describes its purpose as managing crawler traffic so that your server is not overloaded, and states that it is not a mechanism for keeping a web page out of Google (Google Search Central: introduction to robots.txt). A disallowed URL can still be indexed from links on other sites and shown without a snippet. To keep a page out of results, use a noindex meta tag or password protection, and leave the page crawlable so the tag can be read; the Meta Tag Generator writes that tag.

The format is standardised in RFC 9309: groups start with one or more User-agent lines, followed by Allow and Disallow rules whose paths begin with /. Field names are case-insensitive, path values are case-sensitive, and when several rules match a URL the longest one wins. A crawler picks the group whose user-agent token matches it most specifically and ignores the others, so a Googlebot group replaces the * group for Google rather than adding to it.

What this generator writes

For each crawler you set to something other than "Same as Default", the tool writes a group for that user agent with either Disallow: / (block everything) or an empty Disallow:, which crawlers ignore, so everything is allowed. It then writes the User-agent: * group: your list of disallowed paths when the default is Allow, or Disallow: / when the default is Disallow. A path typed without a leading slash gets one added. If you enter a sitemap address, a Sitemap: line is appended; the Sitemaps protocol defines that line so crawlers can find the sitemap without a separate submission.

Some things it does not do. It writes no Allow lines, so you cannot re-open a subfolder inside a blocked one here; add that line by hand. It does not write wildcards (* and $) or Crawl-delay, which Google ignores in any case (How Google interprets the robots.txt specification). When the default is set to Disallow All, the path list is ignored because everything is already blocked. And the four named crawlers are the only ones offered; for any other bot, copy the output and add a group with its token. Everything runs in your browser; nothing you type is sent to our server.

After you upload the file

Save the text as robots.txt in the root of the host, then open https://yourdomain/robots.txt in a browser to confirm it is served as plain text with a 200 status. Google's specification page, linked above, says Google treats a 4xx response other than 429 as if no robots.txt exists and crawls without restrictions, and that it reads at most 500 KiB of the file. Paste the live file into the Robots.txt Validator to catch rules outside a group or a stray Disallow: /, and test an individual URL's crawl and index signals with the URL Checker.

Frequently Asked Questions

Will a page blocked in robots.txt stay out of Google's results?

No. The rule stops Googlebot from fetching the page, but Google can still index the URL from links elsewhere and show it in results without a description. To keep a page out of results, add a noindex meta tag or password-protect it, and do not block the URL in robots.txt, or the tag will never be read.

Why add the Sitemap line?

It tells any crawler that reads your robots.txt where the XML sitemap lives, without you submitting it in each search engine's console. Use the full URL, including https://.

What happens if I don't have a robots.txt file?

Google treats a 404 response as permission to crawl everything, so a missing file is not an error. Add one when you have paths you want crawlers to skip, such as search result pages or admin folders, or when you want to publish your sitemap address.