A store owner opens the server logs and finds visitors called GPTBot, ClaudeBot and PerplexityBot alongside Googlebot. The hosting company's blog says to block them all. A consultant says to add a file called llms.txt instead. Both are half right. Which crawlers you allow decides whether an assistant can read your site at all, and llms.txt is a courtesy that takes ten minutes and promises less than its fans claim.
Three kinds of AI crawler, doing three different jobs
The companies behind the major assistants publish the names of their crawlers and what each one is for, and the names fall into three groups.
- Training crawlers collect pages to train future models. OpenAI documents GPTBot for this, Anthropic documents ClaudeBot, and Common Crawl documents CCBot, whose public archive many models are trained on.
- Search-index crawlers build the index an assistant searches when it answers. OpenAI documents OAI-SearchBot for ChatGPT's search, Anthropic documents Claude-SearchBot, and Perplexity documents PerplexityBot.
- User-triggered fetchers fetch a page because a person asked the assistant about it or clicked through. OpenAI documents ChatGPT-User, Anthropic documents Claude-User, and Perplexity documents Perplexity-User.
Google is the odd one out. Its crawler documentation says AI Overviews and AI Mode are part of Google Search and use the ordinary Googlebot and index; there is no separate crawler to allow. Google-Extended isn't a crawler either. It is a token you can name in robots.txt, and Google documents that it controls whether your content is used to train and ground Gemini, and that it has no effect on Search, AI Overviews included.
Crawler names change as products change, so check each company's current documentation before you edit a rule.
Decide what to allow
The citation path runs through the second and third groups. If you want ChatGPT to cite you, OAI-SearchBot and ChatGPT-User need to reach your pages. Blocking the training crawlers is a separate decision about whether your content may train future models, and because the companies publish separate names, you can make the two decisions separately: block GPTBot and allow OAI-SearchBot, for instance.
In robots.txt that looks like this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
The accident to avoid is the blanket rule. A Disallow: / under User-agent: * left over from a site launch blocks every assistant's crawler along with Googlebot, and a hosting plugin's "block AI bots" switch often blocks the search crawlers as well as the training ones.
Two more things hide outside robots.txt. Robots.txt is a request rather than a lock: Google's documentation is clear that it isn't a way to keep pages out of Google, and the same is true of every other crawler, so enforcement happens at your firewall or CDN. And some CDNs now block AI crawlers with one switch; Cloudflare has said in its own announcements that new domains get that block turned on by default. A site can be invisible to every assistant without a single line in its robots.txt, so check the CDN settings too.
Finally, most AI crawlers fetch the raw HTML and don't run JavaScript the way Googlebot does. That is observed rather than documented, but it means content that only appears after scripts run is content those crawlers never see.
What llms.txt is, and what it isn't
llms.txt is a proposal, not a standard. Jeremy Howard of Answer.AI published it in September 2024 at llmstxt.org. It asks site owners to place a Markdown file at /llms.txt containing a heading with the site's name, a short summary, and sections that list the pages that matter with a one-line description each. An optional /llms-full.txt holds the full text of those pages. The idea is to hand an assistant a clean map of the site instead of making it parse navigation-heavy HTML.
What it doesn't do matters more. It isn't a robots file, so it can't allow or block anything. No major assistant documents reading it, and Google staff have said publicly that Google doesn't use it. It has no effect on indexing or ranking in Google. Its effect on citations is unproven, and anyone who tells you otherwise is guessing.
Where it is used today is narrower: developer documentation sites, and tools that let a person feed a whole site to an assistant directly.
So why bother at all. It's free, it's harmless, it takes minutes, it gives you a plain summary of your business in your own words, and some AI-readiness checks look for it. Just don't pay for it, don't expect rankings from it, and don't let it stand in for the work that matters: crawlable HTML, self-contained passages, and mentions on other sites.
Make sure the crawler can read what it fetches
Allowing a crawler in is only useful if the page it fetches is readable. Serve your content as HTML on first load rather than assembling it in the browser. Keep response times reasonable, because AI crawlers give up on slow pages sooner than Googlebot does. And make sure the crawler sees the same page a person does: the fix for JavaScript-rendered sites in the Sitemaps, robots.txt and indexing lesson helps Google and the assistants alike.
What to take away
- AI crawlers come in three documented kinds, training, search-index and user-triggered, and citations depend on the second and third.
- You can block training crawlers and still allow search crawlers by naming each one separately in robots.txt; the blanket rule and the CDN switch are where sites lock everyone out by accident.
- Google's AI Overviews and AI Mode use ordinary Googlebot, and Google-Extended only controls Gemini training, not Search.
- llms.txt is a cheap, harmless proposal that no major assistant documents reading; add it, but don't expect it to move anything.
Next
With the crawlers let in and the file in place, find out whether any of it is working in Measuring your AI visibility.