ClaudeBot
ClaudeBot is Anthropic's training crawler, described in Anthropic's crawler documentation as collecting web content that could contribute to training its generative AI models. It sits alongside two other agents: Claude-User, which fetches pages in response to a Claude user's question, and Claude-SearchBot, which navigates the web to improve search result quality. As with OpenAI, the training crawler and the search crawler are separate controls.
The three agents
| Agent | What Anthropic says it does |
|---|---|
ClaudeBot | Collects web content that could potentially contribute to model training |
Claude-User | Accesses websites when an individual asks Claude a question |
Claude-SearchBot | Navigates the web to improve the relevance and accuracy of search responses |
The structure mirrors OpenAI's split, and the same reasoning applies: ClaudeBot is a training decision, Claude-SearchBot is a retrieval decision, and Claude-User shows up because a specific person asked something.
Each is a distinct robots.txt token. Blocking ClaudeBot does not block the other two.
Why Claude-User traffic is worth watching
User-triggered fetches are the most commercially legible line in an AI crawler log. Claude-User, ChatGPT-User and Perplexity-User hits mean a person, mid-task, is asking an assistant about that specific URL or that specific product.
In distribution that pattern tends to cluster around exactly the pages you would hope: spec sheets, cross-reference tables, compatibility pages. It is worth breaking out in log analysis separately from scheduled crawler traffic, because volume on a page is a signal about which of your content is being used to answer real buying questions.
Configuration
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Allow: /
Anthropic documents these agents publicly and states that its crawlers respect robots.txt. As with every other AI agent, allowing access is only half the job: if your product pages render client-side, the crawler receives a shell and your specifications never enter any index.
The broader configuration question, covering every major AI agent at once, is worked through in AI crawler.
Frequently asked questions
What is ClaudeBot?
Anthropic's training crawler. Per Anthropic's documentation it collects web content that could contribute to training its generative AI models. It is separate from Claude-SearchBot, which supports search quality, and Claude-User, which fetches pages in response to a user's question.
Does blocking ClaudeBot stop Claude from citing my site?
No. ClaudeBot governs training crawls. Search retrieval is handled by Claude-SearchBot and user-initiated fetches by Claude-User, each with its own robots.txt token. Blocking the training crawler alone leaves the retrieval paths open.
Do Anthropic's crawlers respect robots.txt?
Anthropic documents its crawlers publicly and states that they respect robots.txt directives. Because user-agent strings can be forged by unrelated scrapers, verify by IP where your bot management supports it rather than trusting the string alone.
Is Claude-User traffic worth tracking separately?
Yes. User-triggered fetches indicate a real person asked an assistant about that page, which makes them a demand signal rather than background crawl volume. Watching which product pages attract them tells you which content is being used to answer live buying questions.