"Crawl budget" gets thrown around as a vague warning — don't waste it — without much explanation of what it actually is. Strip away the mysticism and it's simple: a search engine allocates a roughly fixed amount of crawling effort to your site per day, not a fixed number of pages. Effort is spent as time and server load, not as a page count. Which means the same daily budget covers a lot more pages on a fast site than a slow one — and a slow site can look under-indexed for reasons that have nothing to do with its content.
Crawl rate responds to how your server behaves
Google has documented this directly for years: crawl rate — how many requests per second a crawler is willing to make — adjusts based on how your server responds. Answer quickly and reliably, and the crawler is willing to fetch more, faster. Start responding slowly, timing out, or throwing server errors (5xx), and the crawler backs off to avoid making things worse for your visitors — the same adaptive-pacing instinct RacingRoach's own rate control applies inside a single crawl, honoring Retry-After and widening delays after 429/503s instead of hammering a struggling host.
The practical effect: TTFB (time to first byte — how long the server takes to start responding, before any content arrives) is a crawl-efficiency lever, not just a user-experience metric. Shave it down and the same daily crawl budget stretches over more URLs. Let it creep up and fewer of your pages get (re)crawled per day — which means new pages take longer to get discovered, and edits to existing pages take longer to get picked up in search results.
Crawl budget isn't spent per page. It's spent per second. A slow page just spends more of it.
The arithmetic, worked out
This isn't a number we measured — it's illustrative arithmetic to make the mechanic concrete. Assume a search engine allocates a fixed 10-minute crawl window to a site per day (600,000ms), and every request in that window takes roughly the site's average response time. Divide one into the other and you get pages crawled per day at three response-time averages:
Real crawl-budget allocation is more nuanced than one fixed daily window — it also weighs a site's overall demand and how often its content actually changes — but the ratio holds regardless of the exact numbers a search engine uses internally: double the average response time and you roughly halve the pages a fixed time budget can cover. That ratio compounds badly on the sites where crawl budget matters most — the ones with tens or hundreds of thousands of URLs, where a slow average response time doesn't cost you a few missed pages, it costs you weeks of staleness across the whole site.
Finding what's actually slow
The fix starts with knowing which pages are slow, not guessing. A site-wide "average load time" from a synthetic tool tells you nothing about which template, path, or third-party call is the problem — you need response time broken out per URL, across the whole site, from an actual crawl.
That's exactly what RacingRoach's Issues screen surfaces: response-time bands computed across every crawled page, so slow responses show up as a flagged opportunity next to the URL, path, and template they belong to — not buried in a spreadsheet you have to pivot yourself. Because the crawl runs through the same bounded, streaming pipeline regardless of site size, you get that per-URL breakdown on a 5,000-page site exactly the same way you get it on a 500,000-page one, and the crawl-overview dashboard gives you the aggregate picture — how many pages fall into which response-time band — while the crawl is still running, so you don't have to wait for it to finish to know where to start looking.
Once you know which paths are slow, the fix is usually not an SEO fix at all — it's a caching layer, a slow database query, or a template that fires an unnecessary API call on every request. But you can't hand that work to an engineer without first knowing where to point them. Run the crawl, sort by response time, and start with whatever's slowest and most heavily linked.