Say a store owner opens the server logs and finds GPTBot, ClaudeBot and PerplexityBot sitting next to Googlebot. The hosting company's blog says to block them all. A consultant says to add an llms.txt file instead. Each has part of it.
Which crawlers you allow decides whether an assistant can read your site at all. The file is a courtesy that promises less than its fans claim. This lesson covers the crawlers first, because they matter more.
Which AI crawlers exist
The companies behind the major assistants publish their crawler names and what each one is for. The names sort into three groups.
- Training crawlers collect pages to train future models. OpenAI documents GPTBot, Anthropic documents ClaudeBot, and Common Crawl documents CCBot, whose public archive many models learn from.
- Search-index crawlers build the index an assistant searches when it answers. OpenAI documents OAI-SearchBot for ChatGPT search, Anthropic documents Claude-SearchBot, and Perplexity documents PerplexityBot.
- User-triggered fetchers load a page because someone asked the assistant about it or clicked through. ChatGPT-User, Claude-User and Perplexity-User are the documented names.
Google doesn't fit the pattern. Its crawler documentation says AI Overviews and AI Mode are part of Google Search and run on the ordinary Googlebot and index, so there's no separate crawler to allow. Google-Extended isn't a crawler at all. It's a token you name in robots.txt, and Google documents that it controls whether your content is used to train and ground Gemini. Search, AI Overviews included, is unaffected by it.
Names change as products change. Check each company's current documentation before you edit a rule.
Decide what to allow
Citations come through the second and third groups. If you want ChatGPT to cite you, OAI-SearchBot and ChatGPT-User must be able to reach your pages. Training is a separate question, whether your content may teach future models, and because the names are separate you can answer the two questions differently. Blocking GPTBot while allowing OAI-SearchBot is a perfectly coherent position.
In robots.txt it looks like this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
The usual accident is a blanket rule. A Disallow: / under User-agent: *, left over from a site launch, shuts out every assistant's crawler along with Googlebot. A hosting plugin's "block AI bots" switch often does the same to the search crawlers when you only meant to stop training.
Two other things sit outside robots.txt. Robots.txt is a request rather than a lock: Google's documentation is explicit that it isn't a way to keep pages out of Google, and other crawlers can ignore it too, so real enforcement happens at your firewall or CDN. Some CDNs also block AI crawlers with a single setting, and Cloudflare has said in its own announcements that new domains get that block turned on by default. Your robots.txt can be spotless while every assistant is still turned away, so look at the CDN dashboard as well.
Most AI crawlers also appear to fetch raw HTML without running JavaScript the way Googlebot does. That's observed behavior, not documented, but it suggests that anything your pages only show after scripts run may never reach them.
What an llms.txt file is
Jeremy Howard of Answer.AI proposed llms.txt in September 2024 at llmstxt.org. You put a Markdown file at /llms.txt with your site's name as a heading, a short summary beneath it, and sections listing the pages that matter, each with a one-line description. An optional /llms-full.txt holds the full text of those pages. The aim is to hand an assistant a clean map so it doesn't have to pick through navigation-heavy HTML.
What it doesn't do
It can't allow or block anything, because it isn't a robots file. No major assistant documents reading it, and Google staff have said publicly that Google doesn't use it, so it has no bearing on how you're indexed or ranked there. Whether it changes citations anywhere is unproven, and anyone who promises a result is guessing.
Where it does get used is narrow: developer documentation sites, and tools that let a person feed a whole site to an assistant by hand.
So is it worth making? Yes, as a small job. It's free and harmless, it forces you to describe your business in your own words, and some AI-readiness checks look for it. Don't pay anyone for it, and don't let it replace crawlable HTML, passages that make sense on their own, and mentions on other sites.
Make sure the crawler can read what it fetches
Letting a crawler in only helps if the page it gets is readable. Serve your content as HTML on first load instead of building it in the browser. Keep responses quick, since AI crawlers give up on slow pages sooner than Googlebot does. And check that the crawler sees what a visitor sees: the JavaScript-rendering fix in the Sitemaps, robots.txt and indexing lesson helps Google and the assistants alike.
Frequently asked questions
What is an llms.txt file?
It's a Markdown file placed at /llms.txt that summarizes your site for AI assistants and links to your most useful pages with a short description of each. Jeremy Howard of Answer.AI proposed it in September 2024. It's a proposal, not an official standard.
Do llms.txt files work?
Not in any way a major assistant has documented. Google staff have said Google doesn't use the file, and its effect on AI citations is unproven. It does no harm and costs little, so it's reasonable to add, but don't expect it to move rankings or citations.
Where do you put an llms.txt file?
At the root of your domain, so it loads at yourdomain/llms.txt. Upload it there, then open that address in a browser to confirm it's served. The free llms.txt generator builds the file from your business details.
Is llms.txt mandatory?
No. Nothing requires it, and no assistant is documented as depending on it. Your robots.txt rules and CDN settings decide whether AI crawlers can read your site, so fix those first.
What to take away
- AI crawlers come in three documented kinds, training, search-index and user-triggered, and citations depend on the second and third.
- You can block training crawlers and still allow search crawlers by naming each one separately in robots.txt; the blanket rule and the CDN switch are where sites lock everyone out by accident.
- Google's AI Overviews and AI Mode use ordinary Googlebot, and Google-Extended only controls Gemini training, not Search.
- llms.txt is a cheap, harmless proposal that no major assistant documents reading; add it, but don't expect it to move anything.
Next
With the crawlers let in and the file in place, find out whether any of it is working in Measuring your AI visibility.