Bytespider
Bytespider is ByteDance's web crawler, used to gather page content for the Toutiao search product and to build training data for ByteDance's AI models. It identifies itself with the user-agent token `Bytespider`, and ByteDance's own guidance says it honors the Robots Exclusion Protocol. Independent monitoring from multiple bot-tracking services has repeatedly found it crawling at high volume with inconsistent robots.txt compliance and no published IP range, which is why sites that want a reliable block typically enforce it at the CDN or edge layer rather than relying on robots.txt alone.
What Bytespider collects and why
ByteDance — the company behind TikTok — operates Bytespider to fetch publicly reachable pages for two purposes: feeding Toutiao, ByteDance's news and content aggregation product in China, and building the training corpora behind ByteDance's own large language models. It identifies itself in the request header as Bytespider, typically inside a fuller user-agent string that also includes a contact reference.
Unlike GPTBot or ClaudeBot, ByteDance has not published detailed, consistently maintained documentation covering IP ranges or a clear scope statement for what Bytespider does with collected content, which several independent crawler-tracking services have flagged as a transparency gap relative to other major AI labs' crawlers.
Robots.txt compliance is unreliable in practice
ByteDance's public guidance states that Bytespider respects the Robots Exclusion Protocol and recommends a standard disallow rule to opt out. In practice, multiple independent bot-monitoring services have reported Bytespider continuing to request disallowed paths, and Cloudflare's own crawler-traffic reporting has repeatedly placed it among the highest-volume bots hitting the open web.
That gap between documented behavior and observed behavior is the operative fact for a site owner: a robots.txt rule is worth adding because it's the documented mechanism, but it shouldn't be treated as a guaranteed block.
Blocking it in practice
The documented opt-out is a standard two-line rule:
User-agent: Bytespider
Disallow: /
For sites that need a block they can actually rely on, that rule is a starting point, not the whole answer. Edge-level enforcement — a WAF rule, a CDN bot-management product, or request-pattern filtering on the origin — is what independent monitoring services recommend as the reliable layer, precisely because it doesn't depend on the crawler choosing to comply.
Frequently asked questions
Does Bytespider follow robots.txt?
ByteDance's own documentation says it does, and recommends the standard disallow rule to opt out. Independent bot-monitoring services have repeatedly observed it requesting disallowed paths anyway, so robots.txt compliance should be treated as inconsistent rather than guaranteed.
What is Bytespider used for?
Two things: feeding ByteDance's Toutiao content and search product, and collecting training data for ByteDance's own large language models. It is not, as far as publicly documented, tied to TikTok's in-app recommendation algorithm, which runs on separate internal systems rather than open web crawling.
How do I block Bytespider if it ignores robots.txt?
Add the standard robots.txt disallow rule as a documented signal, then enforce the block at the edge — a CDN or WAF rule that filters by user-agent string or request pattern, since that layer doesn't depend on the crawler's own compliance.
Is Bytespider the same as other ByteDance bots?
ByteDance operates other crawlers and fetchers for different purposes, but Bytespider is the primary, most widely observed one associated with content collection and AI training. Sites should still check current traffic logs for other ByteDance-associated user-agent strings, since labs periodically add or rename bot identities.