Robots.txt tester
Enter a URL to see if crawlers are allowed to fetch it, and view the site's robots.txt and sitemaps.
Run a full graded audit for the whole site.
Enter a URL to see if crawlers are allowed to fetch it, and view the site's robots.txt and sitemaps.
A robots.txt tester checks whether a specific URL on a website is allowed or blocked by that site's robots.txt file, which contains instructions controlling how search engines and other crawlers access content.
A robots.txt file is a plain text file placed in the root directory of a website (for example, example.com/robots.txt). It contains directives that tell web robots—including search engine crawlers like Googlebot and Bingbot, plus other automated visitors—which pages and resources they can and cannot access on your site.
The file uses a simple text format with specific directives. User-agent specifies which crawler the rule applies to; Disallow tells crawlers not to access a path; Allow (supported by Google) tells crawlers they may access a path even if a broader Disallow rule would block it; Crawl-delay and Request-rate are older directives mostly ignored by modern crawlers; Sitemap points to your sitemap for search engine discovery.
A basic example looks like this: User-agent: * Disallow: /admin/ Disallow: /temp/ Allow: /admin/public/ Sitemap: https://example.com/sitemap.xml This tells all crawlers not to access /admin/ or /temp/, but allows access to /admin/public/, and provides the location of the site's sitemap.
Important: robots.txt controls whether a crawler can fetch a page. It does NOT prevent a URL from being indexed if the page is linked to from somewhere a crawler can reach. If another website links to a blocked URL, search engines may still index it based on that link—they just won't be able to see the page's content. To truly prevent indexing, you need a meta robots noindex tag or an x-robots-tag header instead.
The test uses the official Google robots.txt parsing standard (RFC 9309) to simulate how a search engine crawler interprets the rules, ensuring accuracy and compliance with how real crawlers behave.
When a URL shows as Allowed, it means the crawler can fetch that page if it finds a link to it; there is no robots.txt rule preventing access. When it shows Blocked, the robots.txt file contains a Disallow rule (or other directive) that prevents the crawler from accessing that specific URL pattern.
The most dangerous robots.txt mistakes are those that accidentally block important content. Testing individual URLs before deploying changes prevents these costly errors.
Testing before deployment is the fastest way to catch these mistakes. If you're making changes to robots.txt, test a sample of your top pages, product pages, blog posts, and any other critical URLs to ensure none are accidentally blocked.
robots.txt has a direct, measurable impact on SEO because it controls which pages search engines can crawl and analyze. A misconfigured robots.txt can eliminate your organic traffic; a well-configured one can improve crawl efficiency and rankings.
Crawl budget is the concept underlying robots.txt's importance. Search engines have finite resources and only crawl a certain number of pages per site per day. Large sites, news sites, and sites with many parameter-heavy or duplicate URLs frequently hit crawl budget limits. A well-configured robots.txt blocks wasteful crawls of /admin/, /temp/, /user-settings/, /search-results/ with lots of parameters, and other non-indexable pages. This preserves crawl budget for the pages that matter—your homepage, product pages, blog posts, and other content meant to rank.
If robots.txt accidentally blocks key pages, Google cannot fetch them, cannot see their content, and cannot rank them. A typo that blocks /products/ instead of /products-on-sale/ removes all products from search results. Testing before deploying prevents this catastrophe.
robots.txt also affects discovery speed. Search engines use robots.txt and sitemaps to schedule crawls. If you add a new product section and want Google to crawl it faster, ensuring it's not blocked and adding it to your sitemap accelerates discovery. Conversely, placing a Disallow rule in front of a new section delays its appearance in search results.
For sites concerned about AI training: robots.txt is the standard mechanism to block AI crawlers like GPTBot (OpenAI), Bingbot-Web, Anthropic's crawlers, and others. Testing that these user-agent blocks work correctly is essential if protecting your content from AI training data collection is important to your business.
A critical misconception: robots.txt is one of three separate crawl and indexing control mechanisms, and they work at different stages. Understanding the difference prevents wasted testing and configuration mistakes.
robots.txt controls whether a crawler can fetch a page in the first place. A page blocked by robots.txt cannot be crawled; the crawler never downloads it and never sees its content. This is a hard stop at the fetch stage.
Meta robots tag (in the page's HTML head): <meta name="robots" content="noindex, nofollow"> tells crawlers not to index the page and not to follow its links, even if robots.txt allowed the fetch. The crawler can still download and read the page; it just skips indexing and link-following. This only works for pages the crawler is allowed to reach.
X-Robots-Tag HTTP header works the same way as meta robots but is sent in the HTTP response header instead of in HTML. Same stage: blocks indexing, not crawl.
In practice: robots.txt blocks crawl; meta robots blocks indexing. A page blocked by robots.txt never shows any other directives because the crawler never reaches it. A page allowed by robots.txt but blocked by noindex appears in your Search Console's Coverage report as Crawled but not indexed—the crawler could fetch it, but indexing was refused. You need meta robots or x-robots-tag to prevent indexing; robots.txt alone only controls fetch access.
If you are making changes to your robots.txt file, always test them before uploading to your live site. A mistake deployed to production can reduce organic traffic immediately.
After deploying your robots.txt changes, wait 24–48 hours for Google to refetch your robots.txt file (this can be accelerated using Google Search Console's URL inspection tool). Then check the Coverage report to see if crawl patterns change as expected. Watch for any new URLs appearing in the Blocked by robots.txt category if that was unintended.
If you find that the live site is behaving differently than expected, you can test your current live robots.txt against your URLs using this tester to confirm what is actually blocking them. Sometimes the issue is not robots.txt but a meta robots tag or an x-robots-tag header added by your CMS or security plugin.
robots.txt tells search engines and other crawlers which pages they can and cannot access on your website. It is used to conserve crawl budget by blocking non-essential pages like /admin/, to prevent indexing of duplicate content, to guide search engines toward important pages, and to block AI training crawlers if you wish to opt out of AI data collection. It does not prevent indexing entirely—it only prevents crawling. A URL blocked by robots.txt but linked from another site may still appear in search results without its content.
Use a robots.txt tester to enter any URL on your site and see whether that URL is allowed or blocked by the site's robots.txt rules. Enter the full URL (including https:// and the path), click Test, and the tool will fetch your robots.txt, parse it, and show you whether that URL matches any Disallow or Allow rules. If blocked, it shows which rule blocked it. This is the fastest way to verify your robots.txt configuration is correct before deploying changes and to debug why a page is or is not being crawled.
If a page is blocked by robots.txt and you want it to be crawlable, edit your robots.txt file to remove the Disallow rule that blocks it. For example, if Disallow: /products/ is blocking your products, remove or narrow that rule. If you need to block a broader directory but allow a specific subdirectory, use the Allow rule: User-agent: * / Disallow: /products/ / Allow: /products/public/. After editing, use this tester to verify the URL is now allowed, then deploy the change to your live site. Wait 24 hours for Google to refetch and re-crawl the URL.
Yes, robots.txt is legal and widely recognized as the standard mechanism for controlling crawler access. It is explicitly supported in the Web Content Accessibility Guidelines and is part of RFC 9309, an official internet standard for robots.txt behavior. Using robots.txt to block crawlers is legal; using it to block only specific crawlers (like AI training bots) while allowing search engines is also legal and common. However, robots.txt is voluntary—a malicious crawler can ignore it. For legally enforced access control, use HTTP authentication or server-level IP blocking instead.
Yes, but only if it's linked to from a page that is not blocked. A URL blocked by robots.txt cannot be crawled directly, so its content is not known to the crawler. However, if another website links to that URL, or if it's mentioned in a sitemap, search engines may still include it in search results based on the link or reference—they just won't be able to see the page's content. To prevent both crawling and indexing, use a noindex meta tag or x-robots-tag header in addition to or instead of robots.txt.