AI crawler
An AI crawler is an automated agent that fetches web content on behalf of an AI system, for one of three distinct purposes: building model training data, building a retrieval index for an AI search product, or fetching a page live because a user just asked about it. Those three purposes are governed by different robots.txt tokens from the same vendor, which is why blanket "block AI bots" rules so often produce an outcome nobody intended.
The three jobs
Training crawlers collect content that may be used to train models. GPTBot (OpenAI), ClaudeBot (Anthropic), Amazonbot, Bytespider (ByteDance), Meta-ExternalAgent. The Google-Extended and Applebot-Extended tokens sit in this category as use controls rather than crawlers.
Retrieval crawlers build the index an AI search product answers from. OAI-SearchBot, PerplexityBot, Claude-SearchBot. These are the ones that decide whether you can be cited.
User-triggered fetchers load a page because a person asked a question that required it. ChatGPT-User, Perplexity-User, Claude-User. Some vendors document these as generally ignoring robots.txt, on the reasoning that a human initiated the request.
Same vendor, different tokens, different consequences. Conflating them is the most common configuration error in this area.
Reference table
| Agent | Operator | Job |
|---|---|---|
GPTBot | OpenAI | Training |
OAI-SearchBot | OpenAI | Retrieval for ChatGPT search |
ChatGPT-User | OpenAI | User-triggered fetch |
OAI-AdsBot | OpenAI | Ad safety validation |
ClaudeBot | Anthropic | Training |
Claude-SearchBot | Anthropic | Search quality / retrieval |
Claude-User | Anthropic | User-triggered fetch |
PerplexityBot | Perplexity | Retrieval, honours robots.txt |
Perplexity-User | Perplexity | User-triggered, generally ignores robots.txt |
Google-Extended | Use control for Gemini training and grounding | |
Google-CloudVertexBot | Crawls sites at the owner's request for Vertex AI Agents | |
Amazonbot | Amazon | Improving products and services; may train Amazon AI models |
Applebot-Extended | Apple | Use control for Apple generative model training |
Bytespider | ByteDance | Training |
Meta-ExternalAgent | Meta | Training |
Operators update these. Treat the table as a starting point and confirm against each vendor's own bot documentation before you write a policy on it.
A defensible default for a catalog site
Allow every retrieval and user-triggered agent. Make a deliberate, written decision on training agents. Never block by accident.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
Then audit what is actually happening. Robots.txt is a stated policy; your CDN, WAF, and bot-management rules are the enforcement layer, and they frequently disagree with it. A rate limiter that challenges unknown user agents will block retrieval crawlers no matter what the file says.
Access is the easy half
Most catalogs that are invisible to AI search are not blocked. They are serving nothing useful.
AI crawlers generally do not execute JavaScript — crawl-log analysis found no evidence that GPTBot does. A storefront that injects specifications and JSON-LD client-side returns an app shell to every one of the agents in that table. Login walls do the same thing more deliberately: gating price behind authentication is a normal B2B decision, but gating the specification block alongside it removes the product from AI retrieval entirely.
Check by fetching your own PDP without JavaScript and reading the response. That single test resolves more AI visibility questions than any tool.
Frequently asked questions
Which AI crawlers should I allow?
At minimum the retrieval crawlers, since they determine whether you can be cited: OAI-SearchBot, PerplexityBot and Claude-SearchBot, plus the user-triggered agents ChatGPT-User, Perplexity-User and Claude-User. Training crawlers such as GPTBot and ClaudeBot are a separate, deliberate decision.
Do AI crawlers execute JavaScript?
Generally not. Analysis of large-scale crawl logs found no evidence that OpenAI's GPTBot executes JavaScript, and the same limitation applies broadly. Content injected after page load, including client-side JSON-LD, should be assumed invisible to them.
Is robots.txt enough to control AI crawlers?
It is a request, honoured by the documented agents from major providers and ignored by many scrapers, and some user-triggered agents document that they ignore it by design. Enforcement, if you need it, is done at the CDN or WAF by verified IP range, not by user-agent string.
How do I tell whether AI crawlers are reaching my product pages?
Query server logs by user agent and confirm 200 responses on product URLs, then fetch one of those URLs without executing JavaScript and check whether the HTML contains your specifications and JSON-LD. Access without a usable payload is the more common failure.