This robots.txt tester tells you whether a site's robots.txt lets Googlebot, and every other crawler, fetch a given URL. Enter the page address and it reads the live robots.txt file, finds the rule that decides that path, and shows allowed or blocked with the rule quoted. It also lists the sitemaps the file declares and lets you read the whole file. It is free and needs no signup.
How to use the robots.txt tester
- 1Type the full address of the page you want to test, such as example.com/blog/my-post. The path is what gets tested, so paste the exact page, not only the domain.
- 2Press Check, or hit Enter.
- 3Read Crawl access for your path: one line for Googlebot, one for All crawlers (*), each allowed or blocked with the deciding rule in brackets.
- 4Read Sitemaps to see which sitemap files the robots.txt points to.
- 5Open View robots.txt to read the whole file and find the line you need to change.
When to use it
- A page is missing from Google and you want to rule out robots.txt first.
- You just launched or moved a site and want to be sure the staging rule "Disallow: /" did not come with it.
- Search Console says a URL is blocked by robots.txt and you want to see the exact rule.
- You added a rule to keep crawlers out of a folder and want to confirm it blocks what you meant, and nothing else.
Worked example
A pricing page that crawlers may fetch
We tested our own pricing page.
- URL to check
- https://searchblueprint.io/pricing
- Crawl access for /pricing
- Googlebot: allowed (Allow: /). All crawlers (*): allowed (Allow: /)
- Sitemaps (1)
- https://searchblueprint.io/sitemap.xml
Both lines say allowed, and the rule that decides it is Allow: /. The file has no group just for Googlebot, so Google follows the * group like everyone else. Under View robots.txt, 25 Disallow lines keep crawlers out of app screens such as /billing and /api/. None of them starts with /pricing, so the page is open to crawl.
How the tester picks the rule
- A group for Googlebot, if the file has one, beats the * group for Googlebot.
- Within the group, the longest matching rule wins. Disallow: /blog/drafts beats Allow: /blog for /blog/drafts/post.
- When an Allow and a Disallow rule are the same length, Allow wins.
- * in a rule matches any characters and $ marks the end of the URL, as Google's robots.txt documentation describes.
- A path no rule matches is allowed.
Made by SearchBlueprint
This tester is free with no signup. SearchBlueprint, who make it, sell SEO planning: a plan that maps the searches your customers type to the page that should answer each one, so you know which pages must stay open to crawlers.
Get your free plan