The problem
Most desktop crawlers hold the entire crawl in memory: every page record, every discovered link, every issue, all resident in RAM for the life of the run. That's fine at a few thousand URLs. Point the same tool at a catalog, a publisher archive, or a site with a few hundred thousand pages and you get one of two outcomes — the app crashes partway through, or you're stuck hand-tuning workarounds (splitting the crawl into smaller batches, capping crawl depth, watching a memory graph like it owes you money) just to get through the whole site in one piece.
Do it in RacingRoach
There's no setting to configure for this, because there's nothing to configure. Bounded, streaming-to-disk storage (each page's record commits to an on-disk database as soon as it's fetched, instead of piling up in memory) is the only storage mode RacingRoach ships with by default — a 500-page crawl and a 500,000-page crawl run the identical code path. You point it at a domain and start; there's no "large site mode" to switch on.
Two practical things worth knowing before you kick off a big audit:
- The free tier caps at 1,000 pages per crawl. That's plenty for most marketing sites, but a large-site audit needs a license, which removes the cap entirely — see pricing for the details.
- There's an optional in-RAM storage mode for crawls you know are smaller or temporary, if you'd rather trade the on-disk file for extra speed. It's a real trade-off, not a free upgrade — see how it works for what changes when you use it. The default disk-backed store is the one built for scale; reach for in-RAM only when you don't need the crawl to persist.
Interpret the results
A crawl that takes hours instead of seconds needs a different way to check in on it than scrolling a raw results table. The crawl-overview dashboard gives you a live summary while it runs — pages by status code, top issue kinds, and overall progress — so you can tell at a glance whether a huge audit is healthy or quietly piling up 500s, without digging through individual rows.
You also don't need to hand-tune concurrency to keep from hammering the origin server the way you might with a blunter tool. Adaptive rate control paces requests per host, honors Retry-After, backs off on 429/503 responses, and widens its own delay when a server is struggling — the crawl adjusts itself instead of you guessing at a safe concurrency number up front. Sites with hundreds of thousands of pages are exactly where crawl budget stops being an abstract concept and starts being the thing your crawl behavior needs to respect.
Fix it
Let a large crawl run in the background. Because results stream to disk as each page completes, you don't have to wait for the whole thing to finish before you start triaging — open the issues you already have while the crawl keeps going.
If something interrupts it — a reboot, a crash, closing the laptop lid — reopen the same crawl file and it resumes exactly where it left off. There's no re-crawling from scratch and no lost progress: every page's record, its discovered links, and the frontier's position all commit together, so the file is never left half-written.