robots.txt for AI crawlers: block training, allow search

Table checked on 22 October 2026. Method and authorship: Camillo Dantas. Portuguese version: robots.txt para robôs de IA.

Which AI crawlers exist and what does each one do?

The names below are the tokens you write on the User-agent line. The function column repeats what the company says on its own documentation page, opened on the date above. Pages that carry no date say so.

Two points are easy to miss. First, Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. Blocking it does not remove you from AI Overviews or AI Mode, which run on Search itself. Second, a blocked training crawler tells you nothing about whether a search product will show your page. Training and search are separate permissions.

How do I block training but keep AI search visibility?

Write one group per crawler. A minimal example, built from the tokens above:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

This is an illustration, not a recommendation for every site. Whether to allow training is a business decision about your content, your licensing and your goals. Many publishers block training and allow search. Others allow both. The method on this site, described in The LLM Seeding method, assumes that visibility in AI search is something you choose deliberately, crawler by crawler.

Anthropic adds a detail: ClaudeBot supports the non-standard Crawl-delay directive, and the rule must be repeated for each subdomain. OpenAI states that for search it can take about 24 hours from a robots.txt update for its systems to adjust. Plan for that delay when you test.

What about GPTBot specifically?

GPTBot is the training crawler from OpenAI. To block it from the whole site, use the first group in the example. To block only a folder, replace / with the path, for example Disallow: /members/. To let it read only part of the site, combine Disallow: / with specific Allow: lines. Whatever you choose, keep OAI-SearchBot in a separate group. Blocking GPTBot does not block OAI-SearchBot, and the reverse is also true.

ChatGPT-User needs a different mindset. OpenAI says it is not used for automatic crawling and that, because a person starts the action, robots.txt rules may not apply. The same logic appears in Perplexity's description of Perplexity-User. If you do not want user-initiated fetches on a private area, robots.txt is the wrong tool. Use authentication.

What does robots.txt not do?

It is a request to well-behaved crawlers, not access control. A file at the root of a domain cannot hide a page from a person or from a crawler that ignores it. It also does not remove an already indexed URL: for that you need noindex or password protection. Sensitive content belongs behind login, not behind a Disallow line.

It also does not guarantee how an AI system will behave when it cites you. Allowing OAI-SearchBot makes your pages eligible to appear in ChatGPT search. It does not promise that they will. For measurement, see the checklist, which includes a section on access and a section on tracking.

How do I test the rules?

1. Open https://yourdomain.com/robots.txt in a browser and confirm that it loads with a 200 status, in plain text, at the root of every subdomain you use.
2. Read the file from the top. A crawler follows the most specific group that matches its token, so a group for GPTBot overrides the * group for that crawler.
3. Check your server logs for the tokens above over a week. A crawler you blocked should stop requesting pages after its systems refresh the file, which OpenAI puts at about a day for search.
4. Repeat after any change to the CDN, the firewall or the hosting. Security rules that block unknown user agents can stop a search crawler you meant to allow, and robots.txt will look correct while the server refuses the request.
5. Write down the date of the review. Crawler names and behaviors change, so review the table at least once a quarter.

Common mistake: pasting a list of AI bots from a blog post. Lists age quickly, and a token that does not exist does nothing. Copy tokens from each vendor's documentation, listed below.

Where does llms.txt fit?

It does not replace robots.txt. They solve different problems: robots.txt says who may crawl, while llms.txt is a proposed file that points to your most useful pages. Google states that it does not use llms.txt for Search. See the llms.txt generator and validator for the format and for what is known about who reads it. For the Brazilian setup with examples, the hub offers a guide to robots and AI bots.

Frequently asked questions

Sources and review

The table and the answers use only the vendor pages listed below. Where a page shows no date, the access date is given. Crawler documentation changes without notice. Review it quarterly and after any announcement from the vendor.