Build a valid robots.txt with allow/disallow rules, crawl-delay and sitemap lines. Then test it with the robots.txt tester.
robots.txt
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
A robots.txt generator creates a plain-text file that tells crawlers which parts of your website they can access. You specify user-agent rules, disallow or allow specific paths, set an optional crawl-delay, and add your sitemap URL; the tool outputs the correctly formatted file you deploy to your root directory.
A robots.txt file is a plain-text file that communicates with search engine crawlers and other bots visiting your website. It lives at the root of your domain (example.com/robots.txt) and contains instructions about which content crawlers can access. Search engines like Google, Bing, and others read this file before crawling your site to understand your rules.
When a crawler arrives at your website, it first checks for robots.txt at the root domain. If the file exists and returns a 200 HTTP status code, the crawler reads the directives. If it returns a 404 or is unreachable, crawlers typically assume they can crawl everything unless other blocking mechanisms like meta tags are in place.
Without robots.txt, you have no way to communicate crawl preferences to bots. Every website has a limited crawl budget—the number of pages a search engine will crawl in a given period. If crawlers waste budget on duplicate pages, admin interfaces, or low-value content, they may miss important pages you want indexed. A properly configured robots.txt helps direct crawlers to your most valuable content.
Common reasons to use robots.txt include blocking access to admin pages, login areas, search results pages that create duplicate content, staging environments, private user data, or temporary directories. You also use it to point crawlers to your XML sitemap, which helps search engines discover your important pages more efficiently.
This generator works by accepting your crawl preferences as form inputs, then outputting a correctly formatted robots.txt file. You type the rules you want, and the tool ensures they follow correct syntax so search engines interpret them properly.
The generator produces a plain-text file with no special formatting. You upload it exactly as generated—no modifications needed unless you want to add additional rules later.
Robots.txt files have strict syntax requirements. Crawlers will misinterpret or ignore lines that don't follow the rules, so your file won't work as intended. Understanding these rules helps you avoid common mistakes.
File name and location matter. The file must be named exactly robots.txt (lowercase, no spaces). It must be placed in the root directory of your domain. For https://www.example.com, the URL must be https://www.example.com/robots.txt. You cannot place it in a subdirectory or on a subdomain and expect it to control the main domain.
The Allow and Disallow directives work together to create your crawl rules, but they have a specific relationship. Understanding how they interact prevents accidental mistakes that leave parts of your site exposed or blocked.
Disallow tells a crawler it cannot access a path. Disallow: /admin means crawlers cannot access anything under the /admin directory or any files matching that path pattern. By default, if you don't specify a Disallow for a path, crawlers assume it's allowed.
Allow overrides a broader Disallow. If you write Disallow: /private followed by Allow: /private/public, crawlers will block /private but allow /private/public. This is useful when you need to block most of a directory but permit specific exceptions. The most specific rule wins—a longer, more specific path takes precedence over a shorter, broader one.
Query strings and fragments are not blocked by robots.txt. If you disallow /search, a crawler can still access /search?query=example because the query string comes after the path. If you want to block pages based on query parameters, use a different method like meta robots tags or noindex headers.
The order of rules matters for multiple User-agent groups. Crawlers read from top to bottom and use the first matching group. If you have rules for Googlebot and then rules for all bots (*), Googlebot stops at the first matching group and doesn't continue to the catch-all group.
An empty Disallow (Disallow: with no path) means crawlers can access everything. This is the default behavior even if you don't include the line at all.
Even small mistakes in robots.txt can have large consequences. These are the most common errors that undermine your crawl control.
After generating your robots.txt file, deployment and testing are critical. A file that isn't reachable or contains syntax errors won't protect your site or guide crawlers.
Robots.txt changes take effect immediately once uploaded, but crawlers may not re-fetch the file for several hours. If you urgently need to block something, use a meta robots tag or X-Robots-Tag HTTP header as a supplementary measure.
The crawl-delay directive and sitemap URLs are optional but powerful tools for fine-tuning how search engines interact with your site.
Crawl-delay specifies the minimum number of seconds a crawler should wait between requests to your site. A crawl-delay of 2 means the crawler waits at least 2 seconds after fetching one page before requesting the next. This reduces server load during crawling. Use crawl-delay if your server has limited resources and crawlers cause noticeable slowdowns. However, crawl-delay can also slow down how quickly search engines discover new content, so use it conservatively. Most sites do not need a crawl-delay; instead, use server-side rate limiting or contact your hosting provider about crawler traffic.
The Sitemap directive points crawlers to your XML sitemap file. A sitemap is a list of all your important pages that you want indexed. By providing the URL to your sitemap in robots.txt, you make it easy for crawlers to find all your content without following links across the site. You can list multiple sitemaps. For example, if you have separate sitemaps for blog posts, products, and images, you can add all of them. Always use the full URL, including the protocol (https://).
Sitemaps are one of the most valuable additions to robots.txt. Even if you use Allow and Disallow to control crawling, a sitemap ensures that crawlers discover all your important pages and understand your site structure. Most CMS platforms like WordPress generate sitemaps automatically, usually at /sitemap.xml.
A robots.txt file is a plain-text file placed in the root of your website that contains instructions for search engine crawlers and bots. It specifies which parts of your site they can and cannot access. For example, you might block crawlers from accessing your admin pages, login areas, or duplicate content. The file must be named exactly robots.txt and placed at yourdomain.com/robots.txt.
Robots.txt helps you manage your crawl budget by directing search engine crawlers to your most important content and away from low-value pages like admin sections, duplicates, or staging sites. This ensures crawlers spend their resources on pages you want indexed. It also helps you keep private or sensitive pages out of search results and prevents duplicate content from diluting your SEO value.
Yes, this generator is completely free. You can create, edit, and download as many robots.txt files as you need without paying or signing up for an account.
Yes. Robots.txt is a simple text file, so you can edit it directly on your server using an FTP client, file manager, or command line at any time. You can also use this generator to create an updated version and re-upload it. Changes take effect immediately, though crawlers may take several hours to fetch the new version.
Disallow tells crawlers they cannot access a specific path or directory. Allow overrides a broader Disallow to permit access to specific exceptions. For example, you might use Disallow: /private to block a directory, then Allow: /private/public-files to permit access to one subdirectory within it. The most specific rule takes precedence.
Add a Sitemap directive with the full URL to your XML sitemap file. For example: Sitemap: https://example.com/sitemap.xml. You can list multiple sitemaps if you have separate ones for different content types (blog posts, products, images, etc.). The URL must include the protocol (https:// or http://).
Use crawl-delay only if your server experiences noticeable slowdowns from crawler traffic. A value of 1 or 2 seconds is typical. However, crawl-delay can slow down indexing, so consider server-side rate limiting or contacting your host instead. Most websites do not need crawl-delay. Google may also ignore this directive in favor of settings in Google Search Console.