Glossary

Crawl budget

Crawl budget is the number of URLs on a site that a search engine's crawler can and is willing to fetch within a given stretch of time, set by two separate limits: crawl capacity, how much load the crawler will put on a server without degrading it, and crawl demand, how much the crawler actually wants to revisit and discover on the site. Google says the concept mainly applies to large sites — roughly a million or more pages — or medium sites with frequently changing content; a small catalog with a few thousand well-structured pages rarely needs to manage it. For a large product catalog, the practical risk is faceted navigation: filter-combination URLs multiplying into far more crawlable pages than there are actual products, spending the budget on filter permutations instead of product detail pages.

Crawl capacity and crawl demand

Google frames crawl budget as the combination of two separate limits. Crawl capacity is a ceiling set by the server: how fast a site responds and how many simultaneous requests it can absorb before response times degrade, which caps how aggressively a crawler will fetch even if it wants more. Crawl demand is the crawler's own appetite for a given URL, driven by perceived value, popularity, and how often the content there actually changes — a page that rarely updates and gets little traffic gets crawled less often regardless of how much spare capacity exists.

Size matters here: Google states the concept is primarily relevant for large sites, roughly a million or more pages, or medium sites with content that changes quickly, in the tens of thousands of pages. Most sites, by Google's own guidance, don't need to actively manage crawl budget at all — general technical health is enough.

Why faceted navigation is the usual drain on a catalog

Faceted navigation — letting a shopper filter a category page by color, size, material, and brand in any combination — can generate orders of magnitude more unique, crawlable URLs than there are actual products, since every filter combination is technically its own address. Most of those pages are thin, near-duplicate views of the same underlying inventory.

A crawler that spends its available budget working through filter permutations has that much less left to reach genuinely new or updated product detail pages, which is the content actually worth indexing. Common remedies: canonical tags pointing filter-combination URLs back to the core category page, robots rules excluding low-value parameter combinations from being crawled at all, and returning a real 404 — rather than a 200-status "no results" page — for filter combinations that match nothing, so a crawler doesn't keep revisiting a dead end.

Why this now matters for AI crawlers too

Crawl budget as Google defines it is specifically about Googlebot and the traditional search index, but the same shape of problem applies to the separate AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and similar agents — that fetch pages for model training, a retrieval index, or a live answer to a user's question. Each of these operates with its own finite fetch capacity against a site.

A catalog that wastes Googlebot's budget on filter permutations wastes an AI crawler's budget the same way: a thin or duplicate page fetched instead of a real product detail page is a page an AI answer engine can't accurately cite from, because the content that would answer the question was never the page it reached.

Frequently asked questions

What is crawl budget?

Crawl budget is the number of pages on a site that a search engine's crawler can and will fetch in a given period, determined by crawl capacity (a server-side limit on load) and crawl demand (the crawler's own interest in a URL, based on its value and how often it changes).

Does every website need to worry about crawl budget?

No. Google states that crawl budget is mainly a concern for large sites, roughly a million or more pages, or medium sites with content that changes frequently. A smaller, well-structured site generally doesn't need to actively manage it.

How does faceted navigation hurt crawl budget?

Filter combinations on category pages — color, size, material, brand, in any mix — can generate far more unique crawlable URLs than there are actual products, most of them thin or near-duplicate pages. A crawler spending its budget on those permutations has less left over to reach new or updated product detail pages.

Does crawl budget affect AI crawlers like GPTBot too?

The term itself comes from Google's documentation about Googlebot, but the underlying problem — a crawler with finite fetch capacity hitting a site with far more generated URLs than real content — applies the same way to AI crawlers fetching pages for training data, retrieval indexes, or live answers.

Related terms

See it on your own SKUs.

A 30-minute walkthrough on your categories and your supplier data.

Book a demo