GPTBot
GPTBot is OpenAI's training crawler. OpenAI's own bot documentation describes it as "used to crawl content that may be used in training our generative AI foundation models," and it is a separate agent from OAI-SearchBot, which is what surfaces sites in ChatGPT's search features. Blocking GPTBot does not remove you from ChatGPT search, and allowing it does not get you cited — the two are constantly confused.
The three OpenAI agents, and what each one does
| Agent | Purpose per OpenAI's documentation |
|---|---|
GPTBot | Crawls content that may be used in training foundation models |
OAI-SearchBot | Surfaces websites in search results in ChatGPT's search features |
ChatGPT-User | Fetches pages in response to a specific user action in ChatGPT |
OAI-AdsBot | Validates the safety of pages submitted as ads on ChatGPT |
The distinction that matters commercially is the first two rows. GPTBot feeds training. OAI-SearchBot feeds the retrieval index behind ChatGPT search. They are governed by separate robots.txt tokens, so a blanket User-agent: GPTBot / Disallow: / leaves ChatGPT search access untouched.
Most "block the AI bots" advice published in 2024 predates OAI-SearchBot existing. A robots.txt written from that advice can be simultaneously more restrictive than intended and less.
Should a distributor block it?
It is a genuine judgment call, not an obvious yes or no, and the answer differs by what your content is.
The case for allowing: training exposure is how a model comes to know your brand, your part-numbering conventions, and your product lines without needing to retrieve anything. Models answer plenty of questions from parametric memory. Being absent from that memory is a slow, invisible disadvantage.
The case for blocking: if your differentiated asset is proprietary content — a cross-reference database, curated fitment data, engineering guides you spent years building — training exposure hands it over with no attribution and no traffic. Many publishers block on exactly this reasoning.
The common middle path is allowing OAI-SearchBot and ChatGPT-User while making a deliberate decision about GPTBot. What is not defensible is blocking all three by accident and then asking why you are never cited.
The rendering problem is bigger than the access problem
Allowing the crawler only matters if what it receives contains your data. Analysis of large-scale crawl logs found no evidence that GPTBot executes JavaScript, and the same limitation applies broadly across AI crawlers.
For a catalog running client-side rendering, that means the crawler fetches a page and gets an app shell: navigation, a loading state, no specifications, no JSON-LD Product block, no price. Access is granted and nothing useful is transferred. Checking this takes one command — fetch your own PDP without executing JavaScript and read what comes back. See server-side rendering for the fix.
Configuring it
GPTBot obeys robots.txt and OpenAI publishes IP ranges for verification. A configuration that reflects a deliberate decision rather than an inherited copy-paste looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /internal/
Allow: /
Two practical notes. Verify by IP as well as user-agent string, since the string is trivially spoofed by scrapers. And if you block, expect a lag: robots.txt changes affect future crawls, not what was already collected.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT's search features are served by OAI-SearchBot, a separate agent with its own robots.txt token. Blocking GPTBot leaves ChatGPT search access unaffected, and blocking OAI-SearchBot is what removes you from those results.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot crawls content that may be used to train OpenAI's foundation models. OAI-SearchBot surfaces sites in ChatGPT's search features. ChatGPT-User fetches a page in response to a specific user action, such as a user asking about a URL. Each is controlled separately in robots.txt.
Does GPTBot run JavaScript?
Analysis of large-scale crawl logs found no evidence that it does. In practice, treat anything injected client-side as invisible to it. If your product specifications or JSON-LD appear only after a fetch resolves in the browser, the crawler receives a page without them.
Should B2B distributors allow GPTBot?
It depends on whether your content is a commodity catalog or a proprietary asset. Allowing it builds passive familiarity with your brand and part numbers in the model itself. Blocking it protects genuinely differentiated content, such as a cross-reference database, that you would rather not donate untraceably.