Are all AI crawlers the same?
No. Both OpenAI and Anthropic say they run separate crawlers for separate jobs. One kind collects content that may be used to train models. Another helps show your site in search answers. A third fetches a page because a person asked a question. You can allow one and block another.
What does OpenAI say its crawlers do?
- OAI-SearchBot helps surface websites in ChatGPT's search features. OpenAI says sites that opt out will not appear in ChatGPT search answers, though they can still show as navigational links.
- GPTBot crawls content that may be used to train its generative AI models. Blocking it signals your content should not be used for that.
- ChatGPT-User visits a page when a user asks ChatGPT to. OpenAI says it does not crawl automatically and that, because users trigger it, robots.txt rules may not apply.
OpenAI says each setting is independent, and that a robots.txt change for search can take about 24 hours to take effect.
What does Anthropic say its crawlers do?
- ClaudeBot collects web content that could contribute to training. Blocking it signals your future content should be excluded.
- Claude-User retrieves pages when a Claude user asks a question that needs them. Blocking it may reduce your visibility in user-directed search.
- Claude-SearchBot indexes content to improve search results. Blocking it may reduce your visibility in them.
What should a business decide?
Two separate questions. First: do you want to appear when people ask an AI about businesses like yours? That points to the search and user crawlers. Second: are you happy for your content to be used to train models? That points to the training crawlers. Either answer is legitimate. The mistake is making the choice by accident, for example through a firewall rule nobody remembers adding. We explain that in our robots.txt article.
Governance: who decides?
This is our reading. Whether to let AI systems train on your content is a business decision about your own material, so someone should own it, write it down and review it, the same way you would any policy on your content.
What do these pages not tell us?
- They describe what each company says it does. We cannot verify how a crawler behaves on your site, which is why we test.
- They cover OpenAI and Anthropic only. Other assistants have their own crawlers and rules.
- Crawler names and rules change, so check the current pages.
Is your website ready for AI?
Our free check reads your site the way AI tools do, tests four AI crawlers live and scores nine dimensions out of 100.
Sources
- Overview of OpenAI crawlers · OpenAI developer documentation, undated page. Read by us on 27 October 2026.
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic support, undated page. Read by us on 27 October 2026.
We keep what a source says separate from our reading of it. If a source is from another country or a different date, we say so. We are not giving advice, and nothing here is a promise of rankings, citations or sales.