robots.txt
robots.txt is a plain-text file published at the root of a website — for example, https://example.com/robots.txt — that tells web crawlers which parts of the site they may or may not crawl, using the IETF-standardized Robots Exclusion Protocol (RFC 9309). It is advisory: a crawler that honors robots.txt checks it before fetching other pages, but nothing in the protocol technically stops a crawler from ignoring it. For AI search visibility, robots.txt is the access switch — it decides whether a given AI crawler can reach a page at all, before any question of whether that page's content gets read, indexed, or cited.
How the file actually works
A robots.txt file lives at the root of an origin — scheme, host, and port — and only governs that origin. Rules are organized into groups, each starting with one or more User-agent: lines naming a crawler (or * for all crawlers), followed by Disallow: and Allow: directives for specific paths, and an optional Sitemap: line pointing to the site's XML sitemap.
The protocol was an informal convention from 1994 until the IETF formalized it as RFC 9309 in 2022, codifying syntax, error handling, and caching behavior that had previously varied across crawlers. A compliant crawler fetches and parses robots.txt before crawling other URLs on that origin and caches the result rather than re-fetching it on every request. Compliance itself is voluntary: nothing in the file blocks a request technically, it only states the site owner's request.
One domain, many AI crawlers
Most AI companies now operate at least two categories of crawler per product: a training crawler that harvests content for model training (for example, GPTBot, ClaudeBot, Google-Extended), and a separate retrieval or answer-time crawler that fetches pages live to ground or cite a response (for example, OAI-SearchBot, PerplexityBot). Because these are distinct User-agent tokens, a site can write separate rules for each — disallowing a training crawler while allowing the corresponding answer crawler, or the reverse. Blocking one of a provider's bots does not block that provider's other bots; each User-agent group is evaluated independently.
Compliance is asserted, not guaranteed
Major AI providers publish documentation stating their crawlers honor robots.txt, but enforcement depends entirely on the crawler choosing to comply — there is no technical mechanism forcing it. In 2025, Cloudflare publicly stated that Perplexity's crawler was accessing sites that had disallowed it in robots.txt, describing behavior including rotating IP ranges and browser-like user-agent strings, and removed Perplexity's bot from its list of verified crawlers as a result; Perplexity disputed that characterization. Whichever account is closer to right, the episode is a useful reminder: a robots.txt rule is a request a well-behaved crawler follows, not a wall a crawler cannot get past.
robots.txt vs llms.txt
The two are often confused but do different jobs. robots.txt controls access — which paths a crawler is allowed to fetch at all. llms.txt is a proposed, non-standardized content-curation file that points crawlers already allowed in toward a site's most important pages; it grants no access on its own. robots.txt is the gate; llms.txt, where a crawler honors it, is closer to a curated index for whoever has already been let through.
Where catalog sites get this wrong
robots.txt rules apply at the domain or directory level, not per product — there's no native mechanism to expose one PDP to an AI crawler while hiding another in the same disallowed path. The common failure is a blanket Disallow: / or a disallowed /products/ directory, written to block AI training crawlers, that also blocks answer and retrieval crawlers sharing the same rule or a wildcard User-agent: * group. The practical result is a catalog that's simply absent from ChatGPT, Perplexity, or Google AI Overviews answers, with no way to diagnose it short of reading the file. A deliberate configuration names each crawler explicitly rather than relying on one catch-all group.
Frequently asked questions
What is robots.txt used for?
It tells web crawlers, including AI crawlers, which parts of a site they may or may not crawl. It's published as a plain-text file at the site's root and governs crawler access at the domain or directory level.
Does robots.txt actually stop crawlers from accessing a page?
Only if the crawler chooses to honor it. robots.txt is an advisory protocol — a well-behaved crawler checks it first and respects the rules, but nothing in the standard technically prevents a crawler from ignoring it. Enforcement, where it exists, happens outside robots.txt entirely, through firewalls or bot-management tools.
How do I block AI training crawlers but still allow AI answer engines?
Write separate User-agent groups for each bot, since training and answer/retrieval crawlers from the same company use different tokens — for example, disallowing GPTBot while allowing OAI-SearchBot. A single wildcard `User-agent: *` rule can't make this distinction; it blocks or allows every crawler that matches it.
What's the difference between robots.txt and llms.txt?
robots.txt controls whether a crawler can access a page at all. llms.txt is an unofficial, non-standardized file that points crawlers already allowed in toward a site's most important content — it's a curation aid, not an access control.
Can robots.txt block AI crawlers from individual products but not others?
Not directly. robots.txt rules apply to URL paths and directories, not individual products within an allowed path. Fine-grained, per-product curation for AI crawlers requires something layered on top, such as structured data or a curated llms.txt file, since robots.txt itself only turns access on or off for whole path sets.