Crawl Errors: What They Mean and How to Fix Them Fast
By Toy Wen Da, Co-Founder · · 6 min read · scored 94/100
Crawl errors in Google Search Console mean Googlebot tried to reach your site, or a specific page, and couldn't finish. The cause is usually a DNS hiccup, a server timeout, or an unreachable robots.txt file. Check whether the error is site-wide or limited to one URL first, since that decides the fix.
Before you go digging through Search Console, it helps to rule out robots.txt as the culprit in under a minute. The free Robots.txt Tester checks whether your robots.txt file loads, returns a clean 200 response, and isn't accidentally blocking Googlebot, which is exactly what triggers a "robots.txt unreachable" message. If the file loads fine there, your problem is almost certainly server-side, not the file itself.
What's Actually Causing Your Crawl Errors
Google drew this exact line years ago: site errors versus URL errors. Site errors aren't tied to one page, they're DNS resolution failures, server connectivity problems, and robots.txt fetch issues that stop Googlebot before it even requests a URL. URL errors happen after Googlebot successfully reaches your server and robots.txt, but hits a snag on a specific page, like a 404 or a server error.
| Error type | What it means | Typical cause | Fix |
|---|---|---|---|
| Robots.txt unreachable | Googlebot can't fetch robots.txt at all | Server downtime, firewall blocking Googlebot's IP range, slow response | Confirm the file returns a 200 status and loads fast; whitelist Googlebot if a firewall is involved |
| DNS error | Googlebot can't resolve your domain | DNS misconfiguration, expired nameserver records | Check DNS propagation and nameserver settings with your host |
| Server connectivity | Googlebot connects but times out | Overloaded server, aggressive rate limiting | Check server response times and raise rate limits for Googlebot's user agent |
| 404 / soft 404 | A specific page doesn't exist or returns thin content | Deleted page, broken link, bad redirect target | Restore the page, 301-redirect it, or remove the dead link |
| Server error (5xx) | The server returned an error on that URL | Code bug, database timeout, plugin conflict | Check server logs for that path and fix the underlying bug |
How to Fix Crawl Errors in Google Search Console
Check the robots.txt report first
In Search Console, open Settings, then look for the robots.txt report. It shows the last time Google successfully read the file and flags any fetch failures. If it shows a failure, that confirms a site error, the fix lives on your server or hosting side, not in the file's syntax.
Look at Crawl stats and host status
Still under Settings, open the Crawl stats report. It breaks down crawl requests by response code and shows Host status, which flags DNS, connectivity, and robots.txt problems at a glance. A red flag here means Googlebot is struggling to reach your server generally, not just one page.
Fix individual URL errors
For a single broken page, use the URL Inspection tool. Paste the URL, check its indexing status, and see exactly what Googlebot saw. Once you've fixed the issue, restored the page, corrected a redirect, cleared a server bug, use the inspection tool's request option to ask Google to recrawl it. If you're still not sure why Google isn't crawling certain pages at all, this step-by-step fix for getting Google to crawl your site walks through the broader indexing side of the problem.
When Search Console's Tools Are Enough, and When They're Not
For a single page throwing a one-off 404, Search Console is plenty. Inspect the URL, fix the link or redirect, request a recrawl, done. The free tools cover this well.
Where it falls short is recurring, site-wide trouble. If robots.txt keeps going unreachable on a schedule, or Googlebot's crawl rate keeps dropping, you're looking at a pattern, not an incident. That usually means checking server logs directly, reviewing hosting-level rate limits, or auditing how your CDN handles Googlebot's requests. If Google seems to be backing off your site entirely rather than erroring out, the cause is sometimes a crawl-delay directive Googlebot is ignoring in an unhelpful way, worth ruling out before you assume it's a server issue.
Common Mistakes
- Requesting a recrawl before fixing anything. It just reproduces the same error.
- Treating every 404 as urgent. Some are old pages nobody links to anymore; prioritize ones with traffic or backlinks.
- Blocking Googlebot's IP by accident via a security plugin or firewall rule meant for bad bots.
- Assuming robots.txt syntax is the problem when the real issue is the server not responding to the request at all.
- Ignoring the Host status report and only looking at individual URL errors, missing a site-wide pattern.
FAQs
What does "robots.txt unreachable" mean in Search Console?
It means Googlebot tried to fetch your robots.txt file and failed, the server didn't respond, timed out, or returned an error instead of the file's content. It's classified as a site error because it can block crawling of your whole domain, not just one page.
How long does it take Google to recrawl after I fix an error?
There's no fixed timeline. Google revisits based on a page's crawl priority and your site's overall crawl budget. Requesting indexing through URL Inspection can speed things up, but it doesn't guarantee an immediate recrawl.
Do crawl errors hurt my rankings?
A handful of isolated 404s usually won't. A site-wide error, like a robots.txt that's unreachable for an extended stretch, can stop Google from crawling new or updated content, which does affect visibility over time.
What's the difference between crawl errors and indexing errors?
A crawl error means Googlebot couldn't successfully fetch the page or file at all. An indexing error means Googlebot fetched the page fine but chose not to add it to the index, often due to a noindex tag, duplicate content, or a canonical pointing elsewhere.
Can a crawl error disappear and come back?
Yes. If the underlying cause is intermittent, like a server that only times out under load, the error can clear on a good crawl and reappear on the next one. That pattern is a sign to check server performance, not just the page itself.
Check Your Robots.txt Report Today
Start with the robots.txt report and Crawl stats in Search Console, that's where a "robots.txt unreachable" message traces back to its actual cause. Run the page through the free Robots.txt Tester first if you want a faster read on whether the file itself is the problem before you touch server settings. And if crawl errors keep surfacing across your site rather than on one page, talk to the team and get a report that shows exactly what's blocking Googlebot.
About the author
SearchBlueprint researched, wrote and scored this article with the content pipeline our customers run on their own sites. The score in the byline is what our content engine gave this page before it published. Nothing here is reviewed by hand, which is the point: it is the product's output, published where anyone can check it.