Bytespider
Bytespider is ByteDance's web crawler, used to gather page content for the Toutiao search product and to build training data for ByteDance's AI models. It identifies itself with the user-agent token `Bytespider`, and ByteDance's own webmaster documentation says it honors the Robots Exclusion Protocol and lists ten /24 address blocks it crawls from. Independent monitoring has repeatedly found it reaching disallowed paths and arriving from addresses outside those blocks, which is why sites that want a reliable block typically enforce it at the CDN or edge layer rather than relying on robots.txt alone.
What Bytespider collects and why
ByteDance — the company behind TikTok — operates Bytespider to fetch publicly reachable pages for two purposes: feeding Toutiao, ByteDance's news and content aggregation product in China, and building the training corpora behind ByteDance's own large language models. It identifies itself in the request header as Bytespider, typically inside a fuller user-agent string that also includes a contact reference.
Unlike GPTBot or ClaudeBot, ByteDance documents the crawler only on its Chinese-language Toutiao webmaster site, and that page stops short of a clear scope statement for what happens to collected content. It does list the address blocks the crawler uses — so the common claim that ByteDance publishes no IP ranges is wrong — but there is no signed verification scheme comparable to the reverse-then-forward DNS lookup Google documents for Googlebot. That missing piece, not the missing prose, is what bites when you are trying to tell the real crawler from something wearing its name.
Identifying it in logs
ByteDance's webmaster documentation lists three user-agent variants — one each for desktop, Android and iOS. The desktop string reads:
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36
All three carry the token Bytespider with a capital B and append a contact reference, either the webmaster platform URL or a spider-feedback address. The two mobile variants declare obsolete platform versions — Android 5.0 and iOS 7.1.2 — with build numbers no current handset reports. That detail has a practical consequence: device-classification rules will file the traffic as ancient mobile browsers and quietly distort your mobile-traffic numbers unless you match the bot token before the device rules run.
The ten /24 blocks ByteDance publishes are 110.249.201.0/24, 110.249.202.0/24, 111.225.148.0/24, 111.225.149.0/24, 220.243.135.0/24, 220.243.136.0/24, 220.243.188.0/24, 220.243.189.0/24, 60.8.123.0/24 and 60.8.151.0/24 — 2,560 addresses in total. Traffic carrying the token from outside that set is either ByteDance operating beyond its own documentation or a scraper borrowing the name, and without a signed verification method the user-agent string gives you no way to tell which.
Robots.txt compliance is unreliable in practice
ByteDance's public guidance states that Bytespider respects the Robots Exclusion Protocol and recommends a standard disallow rule to opt out. In practice, multiple independent bot-monitoring services have reported Bytespider continuing to request disallowed paths, and Cloudflare's own crawler-traffic reporting has repeatedly placed it among the highest-volume bots hitting the open web.
The most recent dated measurement comes from TollBit research covering the first half of 2026, reported in August 2026, which put Bytespider among three agents that reached disallowed pages on close to half of the European sites naming them in robots.txt. There is a fair counter-argument: a crawler only stops once it re-fetches your updated robots.txt, so some share of any measured violation is propagation lag rather than defiance.
Either way, the gap between documented and observed behavior is the operative fact for a site owner: a robots.txt rule is worth adding because it's the documented mechanism, but it shouldn't be treated as a guaranteed block.
Blocking it in practice
The documented opt-out is a standard two-line rule:
User-agent: Bytespider
Disallow: /
For sites that need a block they can actually rely on, that rule is a starting point, not the whole answer. Edge-level enforcement — a WAF rule, a CDN bot-management product, or request-pattern filtering on the origin — is what independent monitoring services recommend as the reliable layer, precisely because it doesn't depend on the crawler choosing to comply.
Frequently asked questions
Does Bytespider follow robots.txt?
ByteDance's own documentation says it does, and recommends the standard disallow rule to opt out. Independent bot-monitoring services have repeatedly observed it requesting disallowed paths anyway, so robots.txt compliance should be treated as inconsistent rather than guaranteed.
What is Bytespider used for?
Two things: feeding ByteDance's Toutiao content and search product, and collecting training data for ByteDance's own large language models. It is not, as far as publicly documented, tied to TikTok's in-app recommendation algorithm, which runs on separate internal systems rather than open web crawling.
How do I block Bytespider if it ignores robots.txt?
Add the standard robots.txt disallow rule as a documented signal, then enforce the block at the edge — a CDN or WAF rule that filters by user-agent string or request pattern, since that layer doesn't depend on the crawler's own compliance.
Is Bytespider the same as other ByteDance bots?
No, and the distinction has teeth: a robots.txt rule naming `Bytespider` does not cover ByteDance's other agents. `TikTokSpider` is a separate token that fetches external pages linked from TikTok and Douyin posts, largely to build link previews, and ByteDance operates further agents beyond those two. If your intent is to set a policy for the company rather than for one crawler, enumerate the tokens you actually see in your own logs.