CCBot
CCBot is the web crawler operated by Common Crawl, a nonprofit that has published a free, open monthly snapshot of the web since 2011. Its output — raw and lightly processed page archives, released as WARC/WAT/WET files — has become one of the most widely reused datasets in AI: many large language models, including early GPT and LLaMA generations, trained in part on Common Crawl data, either directly or through a filtered derivative like C4. CCBot identifies itself as `CCBot/2.0`, states that it honors robots.txt (including the Crawl-delay directive), and crawls on Common Crawl's own monthly schedule rather than on demand, so blocking it going forward stops future inclusion but has no effect on pages already captured in an earlier snapshot.
What CCBot actually does
Common Crawl is a nonprofit, not an AI company — it has run a monthly, general-purpose web crawl since 2011, well before AI training data was a phrase anyone searched for. CCBot is the crawler behind it, and each monthly run is published as a freely downloadable archive: WARC files (raw HTTP responses), WAT files (metadata), and WET files (extracted plain text).
No single organization controls what happens to that data after publication. Researchers, startups, and AI labs alike download and reprocess it independently, which is exactly what makes it different from a lab-run crawler like GPTBot or ClaudeBot.
Why it's a broader lever than any single AI lab's bot
Blocking GPTBot stops OpenAI's own direct crawl. Blocking ClaudeBot stops Anthropic's. Neither touches Common Crawl, and neither touches the dozen-plus open pretraining datasets — C4, RedPajama, and others — that are built by filtering and deduplicating Common Crawl's raw output rather than crawling the web themselves.
Blocking CCBot is the closer equivalent to a blanket opt-out, because so much downstream reuse traces back to it rather than to any single company's own infrastructure. It isn't a perfect substitute for blocking every AI crawler individually — a lab that crawls directly, like OpenAI or Anthropic, is unaffected by a CCBot block — but for the class of models trained on open, Common-Crawl-derived corpora, it's the single highest-leverage entry in a robots.txt file.
Controlling CCBot in robots.txt
User-agent: CCBot
Disallow: /
Common Crawl states that CCBot honors the Robots Exclusion Protocol, including the Crawl-delay directive if a site wants to slow rather than fully block it. That control is forward-looking only: a page already captured in a prior monthly snapshot remains in that published archive, and robots.txt changes don't retroactively edit or remove past crawls. The general AI-crawler configuration tradeoffs — server-side rendering, agent-specific rules, distinguishing training crawlers from retrieval agents — are worked through in AI crawler.
CCBot vs a model's own crawler
| CCBot | GPTBot / ClaudeBot / etc. | |
|---|---|---|
| Operated by | Common Crawl (nonprofit) | A single AI company |
| Crawl purpose | General-purpose web archive | That company's own model training |
| Who reuses the output | Dozens of downstream researchers and labs | Only the operating company |
| Blocking it stops | Inclusion in Common-Crawl-derived datasets | Only that one company's direct crawl |
The practical takeaway for a catalog or content team: a robots.txt policy that only lists the named AI labs and skips CCBot is missing the crawler with the widest indirect reach.
Frequently asked questions
What is CCBot?
The web crawler run by Common Crawl, a nonprofit that has published a free, open monthly archive of the web since 2011. Its output is widely reused to build AI training datasets, including by organizations that never crawl the web directly themselves.
Does CCBot respect robots.txt?
Common Crawl states that CCBot honors the Robots Exclusion Protocol, including the Crawl-delay directive. Adding `User-agent: CCBot` / `Disallow: /` to robots.txt is the standard way to stop future crawls.
Is blocking CCBot the same as blocking ChatGPT or Claude's own crawler?
No. CCBot, GPTBot, and ClaudeBot are separate crawlers operated by separate organizations, and each is controlled independently in robots.txt. Blocking CCBot doesn't stop a lab's own direct crawl, but it does cut off the many downstream, Common-Crawl-derived datasets that a single-lab block can't reach.
How often does CCBot crawl?
Common Crawl publishes on a monthly cadence, so CCBot's crawl activity follows that schedule rather than crawling continuously or on demand.
Will blocking CCBot remove my past pages from AI training data?
Not automatically. Blocking CCBot in robots.txt affects future crawls; it doesn't retroactively alter or delete pages already captured in a previously published monthly archive, which can continue to be reused downstream.