A site relaunches on a Friday. The developer copies the staging site's robots.txt, which blocks every crawler, and nobody notices until rankings drain away over the following fortnight. The quieter version is a sitemap listing hundreds of deleted pages and none of the new ones, so Google spends its visits on dead ends.
Three small things decide what Google is allowed to see on your site: the sitemap, the robots.txt file, and the noindex and canonical tags on each page. None of them changes the words on a page, all of them can switch a page off, and every one fails silently.
The sitemap is your list of pages worth indexing
A sitemap is an XML file, usually at /sitemap.xml, listing the URLs you want Google to know about. Google's sitemap documentation treats it as a hint for discovery: it helps Google find pages, especially new or poorly linked ones, but a listing doesn't make Google index anything. The same documentation says Google ignores the priority and changefreq values plugins still write, and uses lastmod only when the dates are accurate.
What belongs in it: the canonical version of every page you want indexed, each returning a 200 status. What doesn't: redirected URLs, deleted pages, pages carrying noindex, parameter variants, and the later pages of paginated archives. Each of those teaches Google your sitemap can't be trusted, so it stops helping the pages that need it.
Google's documentation caps one sitemap file at fifty thousand URLs or fifty megabytes uncompressed; bigger sites split into several files listed in a sitemap index. Most platforms generate a sitemap for you, which is also why so many are full of tag archives nobody meant to publish. Tell Google where it lives by submitting it in Search Console and adding a Sitemap: line to robots.txt.
robots.txt controls crawling, not indexing
robots.txt is a plain text file at the root of your domain listing which crawlers may fetch which paths. Google's robots.txt documentation is explicit that it's a tool for managing crawler traffic, not for keeping a page out of results. A page Google is blocked from fetching can still be indexed, as a bare URL with no description, if other pages link to it.
That creates the usual trap. Because Google can't read a blocked page, it can't see a noindex tag on it either, so blocking and tagging at once hides the instruction to remove it. To get a page out of the index, let Google crawl it and let the tag do the work.
Use robots.txt to keep crawlers out of admin areas, internal search results, checkout flows, and the filter and sort URLs an online store generates. Never block the CSS and JavaScript files Google needs to render your pages, or any page you want to rank. It's also where you allow or refuse AI crawlers, covered in llms.txt and AI crawlers. And password-protect staging sites instead of blocking them; a password can't be copied to production by accident.
noindex and canonical decide what stays in the index
noindex is an instruction in a page's meta robots tag, or in an X-Robots-Tag header, telling Google not to store the page. Google's documentation says it only works when Google can crawl the page. Use it on thank-you pages, login pages and thin archives that exist for navigation, not for searchers.
The canonical tag, rel="canonical", names the preferred URL when several addresses show the same content: with and without a trailing slash, or with tracking parameters. Google's documentation on consolidating duplicates calls it a strong hint, not a command, and Google picks a different canonical when other signals disagree, most often because your internal links and sitemap point at the version you didn't choose.
There is no duplicate-content penalty; Google's documentation describes consolidation, not punishment. The cost is that signals split and Google chooses which version to show. The fix is consistency: every page carries a canonical pointing at itself, and the sitemap, the internal links and the canonicals all name the same URL.
Read the Pages report before you guess
When a page isn't showing up, Search Console's Pages report, under Indexing, tells you why. "Excluded by noindex tag" and "Blocked by robots.txt" are the self-inflicted ones above. "Page with redirect" and "Not found (404)" mean the URL isn't a page any more. "Duplicate without user-selected canonical" means Google found copies and chose one for you.
The two that confuse people are "Discovered, currently not indexed", where Google knows the URL but hasn't fetched it, usually because it's far from the homepage, and "Crawled, currently not indexed", where Google fetched the page and chose not to keep it, usually because it looked thin or too close to something already indexed. The first is a site structure problem; the second is a content problem, covered in How Google ranks a page.
The URL Inspection tool shows what Google has stored for one URL and lets you request indexing after a fix; Google Search Console essentials covers the rest.
What to take away
- A sitemap is your list of pages worth indexing: canonical URLs that return 200 and nothing else, and Google treats it as a hint rather than a command.
robots.txtcontrols crawling, not indexing; a blocked page can still appear as a bare URL, and Google can't see anoindextag on a page it isn't allowed to read.noindexremoves a page from the index, the canonical tag picks the winner among duplicates, and your sitemap, canonicals and internal links should all name the same URL.- Search Console's Pages report names the reason a page isn't indexed; read it before you rewrite anything.
Next
Google can now find and keep your pages; the last piece of plumbing is how fast and stable they feel, in Core Web Vitals explained.