GPTBot
GPTBot is OpenAI's training crawler. OpenAI's bot documentation now describes it as "used to make our generative AI foundation models more useful and safe" — broader than the earlier wording about content "that may be used in training" — and it is a separate agent from OAI-SearchBot, which is what surfaces sites in ChatGPT's search features. Blocking GPTBot does not remove you from ChatGPT search, and allowing it does not get you cited — the two are constantly confused.
The three OpenAI agents, and what each one does
| Agent | Purpose per OpenAI's documentation |
|---|---|
GPTBot | Crawls to make OpenAI's generative AI foundation models "more useful and safe" |
OAI-SearchBot | Surfaces websites in search results in ChatGPT's search features |
ChatGPT-User | Fetches pages in response to a specific user action in ChatGPT |
OAI-AdsBot | Validates the safety of pages submitted as ads on ChatGPT |
The distinction that matters commercially is the first two rows. GPTBot feeds training. OAI-SearchBot feeds the retrieval index behind ChatGPT search. They are governed by separate robots.txt tokens, so a blanket User-agent: GPTBot / Disallow: / leaves ChatGPT search access untouched.
Most "block the AI bots" advice published in 2024 predates OAI-SearchBot existing. A robots.txt written from that advice can be simultaneously more restrictive than intended and less.
OpenAI's own list now lives at developers.openai.com/api/docs/bots — the older platform.openai.com/docs/bots URL redirects there — and the GPTBot description has been reworded since the agent launched, from crawling content "that may be used in training" to the broader goal of making the models "more useful and safe". If you wrote a crawler policy against the old sentence, it is worth re-reading the new one.
Should a distributor block it?
It is a genuine judgment call, not an obvious yes or no, and the answer differs by what your content is.
The case for allowing: training exposure is how a model comes to know your brand, your part-numbering conventions, and your product lines without needing to retrieve anything. Models answer plenty of questions from parametric memory. Being absent from that memory is a slow, invisible disadvantage.
The case for blocking: if your differentiated asset is proprietary content — a cross-reference database, curated fitment data, engineering guides you spent years building — training exposure hands it over with no attribution and no traffic. Many publishers block on exactly this reasoning.
The common middle path is allowing OAI-SearchBot and ChatGPT-User while making a deliberate decision about GPTBot. What is not defensible is blocking all three by accident and then asking why you are never cited.
The rendering problem is bigger than the access problem
Allowing the crawler only matters if what it receives contains your data. Analysis of large-scale crawl logs found no evidence that GPTBot executes JavaScript, and the same limitation applies broadly across AI crawlers.
For a catalog running client-side rendering, that means the crawler fetches a page and gets an app shell: navigation, a loading state, no specifications, no JSON-LD Product block, no price. Access is granted and nothing useful is transferred. Checking this takes one command — fetch your own PDP without executing JavaScript and read what comes back. See server-side rendering for the fix.
Configuring it
GPTBot obeys robots.txt and OpenAI publishes IP ranges for verification. A configuration that reflects a deliberate decision rather than an inherited copy-paste looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /internal/
Allow: /
Two practical notes. Verify by IP as well as user-agent string, since the string is trivially spoofed by scrapers. And if you block, expect a lag: robots.txt changes affect future crawls, not what was already collected.
Verifying a request really is GPTBot
The user-agent string is a claim, not evidence. Anyone can set it, and scrapers do — precisely because GPTBot is the token most likely to be sitting in an allowlist. OpenAI publishes a separate IP range file per agent so you can check:
| Agent | IP ranges |
|---|---|
GPTBot | openai.com/gptbot.json |
OAI-SearchBot | openai.com/searchbot.json |
ChatGPT-User | openai.com/chatgpt-user.json |
OAI-AdsBot | openai.com/adsbot.json |
These files are revised, so treat them as a feed rather than a constant: the gptbot.json revision stamped 2026-09-22 listed 18 CIDR prefixes, while searchbot.json listed 40. Fetch them on a schedule and rebuild the allowlist from them instead of pinning a copy into a WAF rule.
One asymmetry is worth knowing before you write the policy. OpenAI notes that ChatGPT-User is not used for automatic crawling, and that robots.txt rules may not apply to it because the fetch happens in response to something a person did. A robots.txt-only position therefore does not stop a ChatGPT user pulling your page into a conversation — which, for most distributors, is the one case you would least want to block.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT's search features are served by OAI-SearchBot, a separate agent with its own robots.txt token. Blocking GPTBot leaves ChatGPT search access unaffected, and blocking OAI-SearchBot is what removes you from those results.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot crawls content that may be used to train OpenAI's foundation models. OAI-SearchBot surfaces sites in ChatGPT's search features. ChatGPT-User fetches a page in response to a specific user action, such as a user asking about a URL. Each is controlled separately in robots.txt.
Does GPTBot run JavaScript?
Analysis of large-scale crawl logs found no evidence that it does. In practice, treat anything injected client-side as invisible to it. If your product specifications or JSON-LD appear only after a fetch resolves in the browser, the crawler receives a page without them.
Should B2B distributors allow GPTBot?
It depends on whether your content is a commodity catalog or a proprietary asset. Allowing it builds passive familiarity with your brand and part numbers in the model itself. Blocking it protects genuinely differentiated content, such as a cross-reference database, that you would rather not donate untraceably.
How do I verify a request is really from GPTBot?
Match the requesting IP against OpenAI's published range file for that agent — `openai.com/gptbot.json` for GPTBot — rather than trusting the user-agent string, which is trivially forged. The files are revised periodically, so fetch them on a schedule: the revision stamped 2026-09-22 carried 18 CIDR prefixes.