This robots.txt generator writes the file that tells crawlers which paths on your site they may visit. Start from a preset (Allow all, Block all or Block admin/cart), set the user-agent, add Allow and Disallow paths one per line, an optional crawl delay and your sitemap URL, and the file updates as you type. Copy it to the root of your site. It runs in your browser and needs no signup.
How to use the robots.txt generator
- 1Press a preset to start from: Allow all, Block all or Block admin/cart.
- 2Set User-agent. Leave it as * to cover every crawler, or type one name, such as Googlebot. The tool writes one group, so one user-agent per file.
- 3Edit Allow (one path per line) and Disallow (one path per line). Paths start with a slash, such as /admin/.
- 4Leave Crawl-delay (optional) empty unless a crawler is overloading your server. Replace the example address in Sitemap URL(s) with your own.
- 5Press Copy, save the text as robots.txt at the root of your domain, and check it with the robots.txt tester.
When to use it
- You're launching a new site and want a robots.txt that lets Google in and points it at your sitemap.
- Your shop's cart, checkout and filtered pages keep turning up in Google and you want crawlers kept off them.
- A Disallow: / line was left behind after a redesign, and you need a clean file today.
- Search Console's robots.txt report says your file was not found.
Worked example
A shop that keeps crawlers out of the cart
We pressed Block admin/cart and put in our own sitemap address.
- Preset
- Block admin/cart
- Sitemap URL(s)
- https://searchblueprint.io/sitemap.xml
- robots.txt
- User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Disallow: /*? Sitemap: https://searchblueprint.io/sitemap.xml
Every crawler may visit the whole site except the admin, cart and checkout folders. Read the last Disallow line before you copy it: /*? blocks every URL with a question mark in it. That covers filtered and sorted shop pages and internal search results, which is usually what you want, but also any page whose address uses ?id= or similar. If your product pages work that way, delete the line.
What each line does
| Line | What it tells crawlers |
|---|---|
| User-agent | Which crawler the rules below apply to. * means all of them |
| Allow | A path crawlers may visit, even inside a folder you have blocked |
| Disallow | A path crawlers should not visit. Disallow: / blocks the whole site |
| Crawl-delay | Seconds to wait between requests. Bing reads it, Google ignores it |
| Sitemap | The full address of your XML sitemap |
Made by SearchBlueprint
This generator is free with no signup. SearchBlueprint, who make it, sell SEO planning: a plan that shows the searches your site could be found for and the page each one belongs on.
Get your free plan