How Googlebot works
Googlebot finds new URLs mainly by following links from pages it has already crawled, and from sitemaps you submit. Google indexes the mobile version of most sites, so most of the requests come from the smartphone crawler. Google says it shouldn't visit most sites more than once every few seconds on average, and there is no fixed crawl schedule.
What it sees on your pages
Google renders pages with a recent version of Chrome and runs the JavaScript it finds, so content that loads with JavaScript can be seen, though rendering can wait in a queue. Googlebot reads the first 2MB of an HTML page or other supported file (64MB of a PDF), measured uncompressed, and each CSS and JavaScript file it fetches gets its own 2MB. Anything past that cut-off is not sent for indexing.
It does not see pages behind a login, or URLs your robots.txt file blocks.
Controlling Googlebot
Crawling and indexing are separate, and Google gives a different control for each:
- To stop it crawling a URL: a Disallow rule in robots.txt.
- To keep a page out of search results: a noindex tag. Blocking the crawl alone doesn't do that.
- To keep a page from everyone: password protection.
- To slow it down: Google lets you reduce the crawl rate if your server can't keep up.
Is it really Googlebot?
Other bots often fake Googlebot's user agent. Google's advice is to run a reverse DNS lookup on the visitor's IP address, which for Googlebot resolves to a googlebot.com name, or to match the IP against the ranges Google publishes.