Applebot
Applebot is Apple's web crawler, gathering the page content that powers Siri web answers, Spotlight Suggestions, and Safari's search features. It identifies itself with "Applebot" in its user-agent string, respects robots.txt, and adjusts its crawl rate automatically rather than honoring a Crawl-delay directive. Applebot-Extended, added in 2024, is a separate, independent user-agent that controls whether content already gathered can be used to train Apple's generative AI models, including Apple Intelligence — disallowing it doesn't remove a page from Siri, Spotlight, or Safari search, and allowing standard Applebot doesn't automatically allow Applebot-Extended.
What Applebot crawls for
Applebot fetches and indexes web pages to power search-adjacent features across Apple's own products: Siri's web answers, Spotlight Suggestions on iOS and macOS, and Safari's search suggestions. It follows standard robots.txt directives, and its user-agent string includes "Applebot" alongside device and OS version details. Unlike some crawlers, it doesn't honor a Crawl-delay directive — instead it adjusts its request rate automatically based on how a site responds, backing off when a server slows down or starts returning errors.
Applebot-Extended and Apple Intelligence
Applebot-Extended is a separate user-agent Apple introduced in 2024, specifically to govern whether content is eligible for training Apple's generative AI models, including the on-device and Private Cloud Compute systems behind Apple Intelligence. It's a control signal layered on top of Applebot's crawl rather than a second crawler making its own requests — disallowing Applebot-Extended in robots.txt opts a site's content out of AI training use without pulling it from Siri, Spotlight, or Safari search results, since those still run on standard Applebot access.
Controlling Applebot in robots.txt
The two user-agents are independent toggles, and getting that wrong is the most common mistake:
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /
That configuration keeps pages crawlable for Siri, Spotlight, and Safari search while opting out of AI-model training use. Blocking Applebot entirely removes a site from all three surfaces; blocking only Applebot-Extended leaves search access untouched.
Why it matters for product pages
A distributor or manufacturer's product pages need to be reachable by Applebot for their specs and content to surface in Siri answers or Spotlight product lookups on Apple devices — the same reasoning that applies to GPTBot, OAI-SearchBot, ClaudeBot, and Google-Extended in the broader AI-crawler landscape. Blocking Applebot outright, whether deliberately or through an overly broad robots.txt rule aimed at bots in general, removes that surface entirely rather than just opting out of AI training.
Frequently asked questions
Does blocking Applebot-Extended also block Applebot?
No. They're independent user-agents with independent robots.txt rules. Disallowing Applebot-Extended stops content from being used to train Apple's generative AI models but has no effect on standard Applebot crawling for Siri, Spotlight, or Safari search.
Does Applebot respect Crawl-delay?
No. Applebot ignores a Crawl-delay directive and instead adjusts its own request rate automatically, slowing down when a site's server shows signs of strain rather than following a fixed delay you set.
What features does Applebot power?
Siri's web answers, Spotlight Suggestions on iOS and macOS, and Safari's search suggestions all draw on content Applebot has crawled.
Should distributors and manufacturers allow Applebot?
Generally yes, if visibility in Siri and Spotlight product lookups on Apple devices matters to the business. Allowing standard Applebot doesn't obligate a site to also allow Applebot-Extended — a site can permit search crawling while still opting product content out of Apple's AI-training pipeline.