Robots.txt Crawl-Delay: Why Googlebot Ignores It (And What Actually Works)

Robots.txt crawl-delay is a directive that tells a crawler how many seconds to wait between requests to your server. The catch is that Google does not support it at all, so the first job is to find out which bots are actually hitting your site hard and then pick a fix each one will respect.

Before you change anything, run your file through the free Robots.txt Tester. Paste in your URL and it shows which rules apply to which user-agent, including any crawl-delay line that sits in a block Googlebot will skip.

What robots.txt crawl-delay controls

A crawl-delay line sits inside a user-agent block:

User-agent: Bingbot
Crawl-delay: 10

This tells Bingbot to wait 10 seconds between hits. It is a blunt instrument that has been informally part of the Robots Exclusion Protocol for years, though it was never standardized across every crawler.

Googlebot does not read or act on it, so if Google is the crawler straining your server, the directive does nothing for you. Bing and Yandex do respect it, which makes it useful when those are the bots causing load. DuckDuckGo leans heavily on Bing's index and rarely crawls independently at the same scale.

Crawler	Supports crawl-delay?	What actually controls its rate	Note
Googlebot	No	Automatic, based on server response and site health	Google's own documentation states it does not use the crawl-delay field
Bingbot	Yes	The numeric value you set, in seconds	Treated as a minimum wait between requests
Yandex	Yes	Same crawl-delay field	Interpreted literally as seconds
DuckDuckBot	Inconsistent	Mostly relies on Bing's crawl data	Rarely a major source of server load on its own

How to fix crawl-delay and wildcard rules on your own site

Work through these steps in order:

1. Find your file. Open yoursite.com/robots.txt in a browser. It must sit at the domain root, and a copy placed in a subfolder won't be read.
2. Locate the editor. On most CMS platforms it lives under a Settings or SEO menu, often labeled "Robots.txt" or "Crawler Access." On a custom-coded site it is usually a static file your developer edits and deploys.
3. Check for crawl-delay lines under Googlebot. Delete any you find, since they do nothing and only clutter the file for the next person who edits it.
4. Audit your wildcard rules. A line like Disallow: /*.pdf$ blocks every PDF, which is usually intentional. But Disallow: /category* can quietly block every URL that begins with that string, pages you wanted indexed among them, so read each wildcard rule the way a crawler would, literally.
5. Save, then re-test the live file with the Robots.txt Tester rather than trusting your memory of what you typed.
6. If you need a clean file from scratch, the Robots.txt Generator produces valid syntax without the stray wildcard mistakes that come from hand-editing.

When your site's own tools are enough and when they aren't

Checking a single URL against your robots.txt takes seconds with any decent tester: paste the URL and see whether it is allowed or blocked.

A recurring or site-wide problem is another matter. If pages are dropping out of search results across a whole section, or a developer pushed a new robots.txt during a deploy and nobody caught it, you are no longer debugging one line but auditing a pattern. That calls for a full crawl of the site compared against the file.

• For one URL and a one-time check, a free tester is enough.
• For a whole-site audit, recurring monitoring, or a site whose robots.txt has changed without sign-off, you need ongoing technical monitoring.

Common mistakes with crawl-delay and wildcards

• Adding a crawl-delay under User-agent: * and expecting it to slow Google down.
• Writing a wildcard meant for one folder that also matches a dozen others sharing the same prefix.
• Forgetting the $ end-of-string anchor, so Disallow: /checkout also blocks /checkout-help and /checkout-faq.
• Letting a developer edit the live file during a deploy with no review step.
• Treating robots.txt as a security measure when it only discourages polite crawlers and doesn't restrict access.

FAQs

Does robots.txt crawl-delay work for Google?

No. Google has stated plainly that it does not support the crawl-delay field in robots.txt. If Googlebot is overloading your server, use Search Console or server-level rate limiting instead.

Which search engines do respect crawl-delay?

Bing and Yandex both honor the directive as a minimum number of seconds between requests. Google and most modern AI crawlers do not.

What's the difference between Disallow and crawl-delay?

Disallow blocks a crawler from a URL path entirely. Crawl-delay blocks nothing and only asks a cooperating crawler to slow down.

How do wildcards work in robots.txt?

The asterisk (*) matches any sequence of characters, and the dollar sign ($) anchors a rule to the end of a URL. Combined carelessly, they are the most common cause of accidental over-blocking.

Can a wrong robots.txt rule deindex my pages?

Yes. If a wildcard disallow rule matches pages that were previously indexed, search engines will stop crawling them and eventually drop them from results.

Get your robots.txt checked properly

Run your live file through the Robots.txt Tester before you assume a crawl-delay or wildcard rule is behaving as intended, because most problems in this file come down to one misplaced character. If you want a second set of eyes on how your whole site crawls, talk to the team and get a sample report on what is blocking your pages.

Resources / References

1. Google does not support it at all
2. Robots Exclusion Protocol
3. Bing
4. Google's own documentation

Sources

• Robots.txt Guide: Essential Rules & Disallow Best Practices
• Can I set Cawl-delay in robots.txt to less than one second ...
• Crawl Delay - robots.txt Directive and Request Pacing