Bingbot
Bingbot is Microsoft's web crawler — the automated agent that discovers and indexes pages for Bing Search, and, because Microsoft Copilot grounds many of its web-sourced answers in the Bing index, for Copilot's citations too. It identifies itself with the user agent `Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)`, respects robots.txt including the Crawl-delay directive, and crawls from IP ranges Microsoft publishes for verification. Blocking it removes a page from Bing results and can reduce how often Copilot surfaces or cites that page, since Copilot's web grounding draws on the same index Bingbot builds.
What Bingbot does, and why Copilot depends on it
Bingbot crawls the web to build the Bing search index — the same core function Googlebot serves for Google. What makes it matter beyond Bing's own market share is downstream reuse: Microsoft Copilot relies on Bing search for many of its web-grounded answers, so a page absent from the Bing index is also less available for Copilot to cite. The same index additionally underpins Yahoo Search and parts of DuckDuckGo's results, through licensing arrangements, so Bingbot access has reach beyond Bing's own traffic share.
Verifying genuine Bingbot traffic
Any crawler can claim to be Bingbot in its user agent string, so a log line alone isn't proof. Two checks confirm it:
- Reverse DNS. A genuine Bingbot request resolves back to a
*.search.msn.comhostname. - Published IP ranges. Microsoft publishes Bingbot's current IP ranges as JSON at
bing.com/toolbox/bingbot.json, which a log-analysis pipeline can check against automatically.
This matters operationally because a distributor blocking a suspected scraper by user agent string alone risks also blocking a spoofed request that happens to share the string — the IP and DNS checks are what separate the two.
Bingbot vs. the rest of the Microsoft crawler family
| Agent | Purpose |
|---|---|
bingbot | Builds the Bing web index |
BingPreview | Renders a page snapshot for preview features |
AdIdxBot | Crawls for Bing Ads quality and relevance checks |
MicrosoftPreview | A related preview/snapshot agent in the same family |
As with the GPTBot / OAI-SearchBot split and the ClaudeBot / Claude-SearchBot split, these are separate robots.txt tokens with separate purposes — disallowing one doesn't disallow the others.
Configuring robots.txt
User-agent: bingbot
Allow: /
Bingbot respects robots.txt, including a Crawl-delay directive if you set one. As with every other crawler in the AI crawler family, allowing access is only half the job: if product detail pages render their attributes client-side with no server-rendered fallback, Bingbot — like Googlebot — receives a shell and the specifications it would otherwise index never make it into the page it actually crawls.
Frequently asked questions
Does blocking Bingbot affect Microsoft Copilot?
Indirectly, yes. Copilot grounds many of its web-sourced answers in the Bing search index, which Bingbot builds. A page Bingbot can't crawl generally isn't in that index, which reduces the odds Copilot surfaces or cites it — though Copilot's exact retrieval behavior beyond the shared index isn't fully published.
How do I verify a crawler claiming to be Bingbot is genuine?
Check two things: that the request's IP falls within the ranges Microsoft publishes at bing.com/toolbox/bingbot.json, and that a reverse DNS lookup on that IP resolves to a *.search.msn.com hostname. A matching user agent string alone isn't sufficient — it can be spoofed by any scraper.
Does Bingbot respect robots.txt?
Yes, including the Crawl-delay directive if one is set. Blocking it with a standard User-agent: bingbot / Disallow rule removes affected pages from Bing search results and reduces their visibility to Bing-grounded Copilot answers.
Is Bingbot the same crawler that feeds ChatGPT or Perplexity?
No. Bingbot is Microsoft's own crawler, separate from OpenAI's GPTBot/OAI-SearchBot and Perplexity's crawlers. Each company runs its own indexing and training agents, so a robots.txt policy has to address each one individually rather than assuming one block covers all AI-related crawlers.