OAI-SearchBot
OAI-SearchBot is OpenAI's retrieval crawler, described in OpenAI's bot documentation as "used to surface websites in search results in ChatGPT's search features." It is the agent that determines whether your pages can be found and cited when a ChatGPT user asks a question, and it is separate from GPTBot, which crawls for model training. If you want to be citable in ChatGPT, this is the one to allow.
Why this one matters most
Of OpenAI's published agents, OAI-SearchBot is the one with a direct line to whether a buyer sees your product. It builds the index that ChatGPT search draws on. Blocking it means your pages are not candidates for retrieval, regardless of how complete your data is or how much training exposure you have.
It is also the agent most likely to be blocked unintentionally. Robots.txt files written during the 2024 wave of "block the AI crawlers" guidance often use broad patterns, and some WAF and bot-management rules treat unfamiliar user agents as scrapers by default. Checking whether it is currently reaching your product pages is a server-log query, not a theory.
Retrieval crawlers versus user-triggered fetches
Three different behaviours hide behind "AI crawler," and it is worth separating them:
- Index-building retrieval, such as
OAI-SearchBot,PerplexityBotandClaude-SearchBot. Scheduled, broad, builds a persistent index. - User-triggered fetches, such as
ChatGPT-User,Perplexity-UserandClaude-User. One-off, in response to a specific person's question, often on a URL they supplied. - Training crawls, such as
GPTBotandClaudeBot. Feed model training, not retrieval.
The volume profiles differ accordingly. Retrieval crawlers show up as steady background traffic; user-triggered agents arrive in bursts that correlate with actual buyer interest, which makes them a genuinely interesting log signal — someone is asking about you right now.
Allowing it is necessary, not sufficient
The failure most catalogs actually hit is not access, it is payload. AI retrieval crawlers generally do not execute JavaScript. A single-page-application storefront returns an app shell, and the crawler indexes a page with no product on it.
The check takes a minute:
curl -sA "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
https://example.com/product/HB-150-50K | grep -c "application/ld+json"
If that returns zero, or the HTML contains no specification values, the crawler is being served nothing usable. Fixing rendering is worth more than any content change downstream of it.
Robots.txt and verification
User-agent: OAI-SearchBot
Allow: /
OpenAI publishes IP ranges for each agent so operators can verify requests rather than trusting the user-agent string, which anyone can forge. If you run bot management, allowlist by verified IP range rather than by string alone.
One more thing worth doing while you are in there: check that your PDPs are not behind a login, a region gate, or an aggressive rate limiter. Distributors gate price behind authentication routinely, which is a legitimate commercial choice. Gating the specifications alongside it removes the product from AI retrieval entirely, and is usually not a decision anyone consciously made.
Frequently asked questions
What is OAI-SearchBot?
OpenAI's retrieval crawler. Per OpenAI's documentation it is used to surface websites in search results in ChatGPT's search features. It builds the index ChatGPT draws on when answering questions, which makes it the agent that determines whether your pages can be cited.
Should I allow OAI-SearchBot?
If you want your products to appear in ChatGPT's answers, yes. It is a retrieval crawler rather than a training crawler, so allowing it does not imply consenting to model training, which is governed separately by GPTBot.
How do I check if OAI-SearchBot is reaching my product pages?
Query your server logs for the user agent and confirm it is receiving 200 responses on product URLs. Then fetch one of those URLs yourself without executing JavaScript and confirm the response HTML actually contains your specifications and JSON-LD.
Does allowing OAI-SearchBot mean OpenAI can train on my content?
The controls are separate. GPTBot governs crawling for model training, OAI-SearchBot governs search retrieval. You can allow one and disallow the other in robots.txt, and many publishers do exactly that.