Crawling and indexing

What is robots.txt, and can it block AI bots?

robots.txt is a plain-text file at the root of a site that tells search engine crawlers which URLs on the site they may fetch and which they should leave alone.

A small robots.txt file

1User-agent: *
2Disallow: /cart/
3Disallow: /search
4Sitemap: https://example.com/sitemap.xml

Every crawler may fetch everything except the cart and internal search pages, and is pointed to the sitemap.

How it works

Crawlers look for the file at yourdomain.com/robots.txt before they fetch anything else. Each group starts with a User-agent line naming a crawler, or an asterisk for all of them, followed by Disallow and Allow lines for paths. A Sitemap line can point crawlers to your XML sitemap.

What robots.txt does not do

Google's documentation says robots.txt is used mainly to avoid overloading your site with requests, and that it is not a way to keep a page out of Google. A blocked URL can still be indexed if other pages link to it; Google just can't read what is on it. To keep a page out of search results, use a noindex tag or put it behind a password.

robots.txt and AI search

Google says robots.txt rules for Googlebot are also how you control crawling for its AI features in Search, since they are part of Search. A separate token, Google-Extended, covers AI training and grounding in some of Google's other systems. Other AI companies publish the names of their own crawlers, which you can give their own group.

SEO glossary

Every term starts with a one-sentence answer and links the free tool or lesson that helps next.

All terms

robots.txt: questions people ask

Where does robots.txt go?

At the root of the host, so it loads at yourdomain.com/robots.txt. Each subdomain needs its own file.

Does every site need a robots.txt file?

No. With no file, crawlers assume they may fetch everything. You need one when there are paths you want crawlers to skip.

Can robots.txt block AI bots?

It can ask them to stay out: give an AI company's crawler its own User-agent group with Disallow: /. The file is a request that well-behaved crawlers follow, not a lock, and blocking Googlebot also removes you from Google Search.

Will blocking a page in robots.txt remove it from Google?

No. Google says to use noindex or password protection to keep a page out of its results.

See which searches your own site should answer next.

Start with the plan for your own site: which page targets each search, and what to write next. The first one is free.

Free first plan
No credit card
Keep your plan