Crawl budget is the combination of two limits a search engine sets for your site: how many requests it's willing to make without overloading your server (the crawl rate limit), and how much crawling it actually wants to do given your site's size and how often it changes (crawl demand). Together they cap how many URLs get crawled in a given window — and every URL a crawler spends on a redirect chain, a duplicate, or a broken link is one it didn't spend on a page you actually want indexed.
Who this actually matters for
For a 40-page marketing site, crawl budget is a non-issue — Googlebot will get through the whole thing in one pass with room to spare. It becomes a real constraint at scale: e-commerce catalogs with faceted navigation generating thousands of filter-parameter URLs, large publishers with years of archived content, or any site where server response time is slow enough that the crawler backs off before finishing. The bigger and slower the site, the more crawl budget behaves like a finite resource instead of an assumption you can ignore.
What wastes it
Crawl budget disappears into a handful of predictable places:
- Duplicate or near-duplicate URLs — the same content reachable through multiple parameter combinations or session IDs.
- Redirect chains — each hop is a separate crawl request before the crawler reaches real content. See redirects for how chains form and how to collapse them.
- Soft 404s and thin pages — pages that return
200but have no real content still cost a crawl. - Slow or error-prone responses — a high rate of
5xxresponses is one of the strongest signals that makes a crawler throttle itself; see HTTP status codes.
Retry-After headers, how often content actually changesHow to see where yours is going
You can't optimize crawl budget without first seeing your site the way a crawler does: every redirect hop, every duplicate cluster, every slow response, counted. That's the same data a full site crawl surfaces — RacingRoach's crawl-overview dashboard gives you a live read on status-code distribution and response-time bands as a crawl runs, so the pages quietly eating your budget show up before a search engine has to tell you by way of a stalled reindex.
Robots directives are one of the few levers you control directly — disallowing low-value URL patterns keeps a crawler from spending budget on them in the first place, which is worth understanding on its own; see robots.txt.