Search Engine Crawling Explained: Why Bots Skip Pages
SearchBlueprint · · 9 min read · scored 92/100
Search engine crawling is the process where bots like Googlebot visit your pages, download their content, and follow the links on them to find more pages. It happens before indexing and ranking, so a page a crawler cannot reach or read never shows up in search results, no matter how good the writing is.
How search engine crawling fits into SEO
Crawling, indexing, and ranking are three separate jobs, and mixing them up is the most common reason site owners misdiagnose a traffic drop. Crawling finds a page, indexing decides whether it is worth storing, and ranking decides where a stored page appears for a given search. A page can be crawled and still never indexed. Each stage has its own controls, and fixing the wrong one wastes time.
| Stage | What happens | What controls it | What blocks it |
|---|---|---|---|
| Crawling | A bot fetches the page and follows its links | Robots.txt, sitemaps, internal links, server response codes | Blocked robots.txt rules, broken links, slow or erroring servers |
| Indexing | The engine decides whether to store the page | Content quality, canonical tags, noindex directives | Duplicate content, thin pages, a stray noindex tag |
| Ranking | The engine orders indexed pages for a query | Relevance, authority, user experience signals | Weak content, poor internal linking, competing pages |
If you are still fuzzy on how these three stages connect to the rest of the discipline, our guide to what SEO actually is walks through the whole system, not just crawling.
How crawling works, step by step
Search engines break the crawl into distinct stages. Understanding each one tells you exactly where a page can fall out of the process.
Discovery
A crawler starts with a list of known URLs, called seeds, plus a queue of URLs it still needs to visit. It adds new addresses to that queue three ways: by re-crawling known pages and following their links, by reading your XML sitemap, and by picking up URLs submitted through a webmaster tool. A new page with no links pointing to it and no sitemap entry can sit undiscovered indefinitely.
Fetching and parsing
Once a URL comes up in the queue, the crawler requests it from your server, the same way a browser would. The server returns the page, usually as HTML, and the crawler parses that code to pull out three things: the links (which go back into the discovery queue), the resources such as images, CSS, and scripts, and the metadata, including the title tag and meta description.
Rendering
Modern pages lean heavily on JavaScript to build their content in the browser. A crawler that only reads raw HTML misses anything JavaScript adds after the page loads, so search engines run a second pass that executes the JavaScript and renders the page much the way a browser does. This step costs more computing time than a simple HTML fetch, which is one reason JavaScript-heavy pages sometimes take longer to appear in search results.
Indexing
If the crawl and render succeed, the page moves to indexing, where the engine stores its content, links, and metadata in a searchable database. Crawling only gets a page to the door. What happens after that is a separate topic, but it starts with everything the crawler managed to read during this stage.
How to recognize a real search engine crawler
Every legitimate crawler identifies itself through a user agent string it sends with each request. That string names the bot and links to a page explaining what it does.
| Search engine | Bot name | User agent string (shortened) | What it crawls for |
|---|---|---|---|
| Googlebot | Mozilla/5.0 (compatible; Googlebot/2.1; ...) | Google Search, Google Images, Google properties | |
| Bing | Bingbot | Mozilla/5.0 (compatible; bingbot/2.0; ...) | Bing Search |
| Yandex | YandexBot | Mozilla/5.0 (compatible; YandexBot/3.0; ...) | Yandex Search |
| Baidu | Baiduspider | Mozilla/5.0 (compatible; Baiduspider/2.0; ...) | Baidu Search |
| DuckDuckGo | DuckDuckBot | DuckDuckBot/1.0; ... | DuckDuckGo Search |
| Apple | Applebot | Applebot/1.0; ... | Siri and Spotlight Search |
Verifying a crawler is real
Anyone can send a request with "Googlebot" typed into the user agent field, including scrapers with no business reading your content. A user agent string alone proves nothing. The reliable check is a reverse DNS lookup on the requesting IP address, followed by a forward lookup that confirms the hostname resolves back to the same address. Google and Bing both publish the IP ranges their crawlers use specifically so site owners can run this check.
Can search engines crawl images, video, and JavaScript pages?
Images and video
Crawlers cannot see a picture or watch a video the way a person does. For non-text files, they read the filename, surrounding text, alt attributes, and any structured data attached to the file. That is a limited signal, but it is enough for the file to be indexed and to rank in image or video search results, even without visual understanding of the content itself.
JavaScript-rendered pages
As covered in the rendering stage above, JavaScript pages are crawlable, but they take an extra step. If your JavaScript framework relies on client-side rendering and something in that process errors out, a crawler can end up with a blank page even though a visitor's browser renders it perfectly. Testing how your rendered HTML actually looks to a crawler, not just to a browser, is the only way to catch this before it costs you a page.
How much of the web actually gets crawled
Crawl coverage has grown enormously, but it has never been complete. A widely cited 2009 study found that even large-scale search engines indexed no more than 40 to 70 percent of the indexable web, up from an earlier study that found no engine indexed more than 16 percent of the web back in 1999. Google itself said in 2016 that it was aware of roughly 130 trillion individual pages, and research from Ahrefs, cited widely across the SEO industry, has found that as much as 96.55 percent of published pages get zero organic traffic from Google. None of that means crawling does not matter. It means the pages that do get crawled and indexed are pulling from a pool where most content never makes it that far.
Google is also not competing on a level field. Once you count Google Images, Maps, and YouTube alongside standard web search, estimates put Google's combined share of search activity above 90 percent, roughly twenty times Bing and Yahoo combined. That is why most crawl troubleshooting starts with Googlebot specifically.
How to control what crawlers can and cannot access
You do not want every crawler reading every URL on your site. Staging environments, internal search results, and filtered product listings are common examples of pages worth keeping out of the crawl entirely.
Robots.txt: the first stop for every crawler
Every compliant crawler checks your robots.txt file, at yourdomain.com/robots.txt, before it requests anything else, per Google's own crawling and indexing documentation. The file lists which paths a crawler may or may not fetch, and it can point to your sitemap location. A misconfigured robots.txt is one of the fastest ways to accidentally block an entire section of a site, so it is worth testing changes before you publish them. SearchBlueprint's robots.txt tester checks your live file against the rules search engines actually follow. If you do not have a robots.txt file yet, the robots.txt generator builds a valid one from scratch.
Sitemaps and URL submissions
An XML sitemap lists the URLs you want crawled, which speeds up discovery on sites where internal linking alone would take too long to reach every page. You can check whether your sitemap is valid and actually reachable with a free sitemap checker before you submit it. For individual pages that need attention faster, most search engines also accept manual URL submissions, though Google specifically caps that method at 10 submissions per day per property, which is why sitemaps remain the better tool for anything beyond a handful of pages.
AI crawlers and llms.txt
Search engine crawling is no longer only about Google and Bing. AI systems such as ChatGPT, Claude, and Perplexity run their own crawlers to gather content for answers and citations, and a growing number of site owners publish an llms.txt file to guide how those bots should read their content. If you have not set one up, SearchBlueprint's llms.txt generator creates one in the same format AI crawlers expect.
How to debug crawling problems
When a page will not show up in search results, work through these checks in order before assuming the content itself is the problem:
- Check robots.txt first. A single disallow rule can quietly block an entire folder.
- Confirm the page returns a 200 status code. A 404, 500, or unexpected redirect stops the crawl cold.
- Look for a stray noindex tag. This blocks indexing even when crawling succeeds.
- Verify the page is reachable by internal links. A page with no internal links pointing to it, sometimes called an orphan page, is much harder for crawlers to find.
- Check server response time. A slow or frequently erroring server teaches crawlers to visit less often.
- Test how the rendered page actually looks to a crawler, not just to a browser, if the page relies on JavaScript.
FAQs
What is the difference between crawling and indexing?
Crawling is discovery: a bot fetches a page and follows its links. Indexing is storage: the engine decides whether that page's content is worth adding to its searchable database. A page can be crawled without being indexed, but it cannot be indexed without first being crawled.
How often do search engines recrawl a page?
There is no fixed schedule. Pages that change often, carry strong links, or sit on high-authority sites tend to get recrawled more frequently, while static or low-authority pages may go weeks or months between visits.
Can I block search engine crawlers from part of my site?
Yes, through robots.txt rules, a noindex meta tag, or password-protecting the section. Robots.txt stops crawling; noindex stops indexing while still allowing the crawl. Use noindex, not robots.txt, for pages you want removed from search results entirely.
Do search engines crawl images and video files?
Yes, but they read filenames, surrounding text, alt attributes, and structured data rather than the visual content itself. That is enough for images and videos to be indexed and to appear in image or video search results.
Why is my new page not showing up in search results yet?
New pages usually need to be discovered before they can be crawled. Submitting your sitemap, linking to the page internally from an already-indexed page, and confirming robots.txt allows it are the fastest ways to speed that up.
Check what is actually blocking your pages
Most search engine crawling problems trace back to one misconfigured file, one broken link, or one status code nobody noticed. Rather than guessing which one applies to your site, look at a sample SEO report showing exactly how a real crawl and index audit is laid out, then talk to the team about running one on your own domain.
Resources / References
Sources
About the author
SearchBlueprint researched, wrote and scored this article with the content pipeline our customers run on their own sites. The score in the byline is what our content engine gave this page before it published. Nothing here is reviewed by hand, which is the point: it is the product's output, published where anyone can check it.