Here's the failure mode we built RacingRoach to avoid: you point a desktop crawler at a site with a few hundred thousand pages, walk away, and come back to a beachballing app and a machine that's swapped itself into the ground. It's not a bug in those tools — it's an architecture. Most crawlers hold the entire crawl, every page record, every discovered link, in memory for the life of the run. That works fine at ten thousand URLs. It stops working at a million.
We took the opposite approach: never hold the full crawl in RAM. Not "try to keep it small" — an actual invariant, enforced through every stage of the pipeline, that we test against directly. The result is a crawl that stays in the hundreds of megabytes at a million URLs instead of the tens of gigabytes a RAM-resident crawler needs.
The pipeline
Every URL moves through four stages, each one bounded so it can only hold so much work before it applies backpressure to the stage behind it:
The parser is the detail people usually get wrong first. A full DOM parse (the kind a browser or a library like jsdom builds) allocates a tree for the entire page and keeps it around for anything downstream that wants to query it. We use lol_html, a streaming HTML rewriter: it fires callbacks as it scans through the byte stream and never materialises a tree. We pull out titles, headings, links, and structured data as the bytes go by, then let the page content go — only the extracted fields and a compact body-text record get stored.
Storage is the other half. Every page's record, its URL-index entry, the frontier cursor advancing past it, and any newly discovered links all commit in one redb transaction. That's not just for memory — it's what makes a crawl crash-safe. Kill the process mid-crawl, or close your laptop lid, and the on-disk file is never left in a half-written state. Reopen it and the crawl picks up exactly where it stopped.
The invariant isn't a setting you switch on for big sites. It's how the pipeline is built — which is also why it's the same code path whether you're crawling 500 pages or 5 million.
What "bounded" actually measures like
We don't like citing round marketing numbers for something this checkable, so here's a synthetic HTTP crawl through the real pipeline (concurrency 16, ~1.5 KB pages, single dev machine) at three sizes, in the default on-disk storage mode:
Two things worth calling out, because they're the parts a reader should be skeptical of by default:
- Memory isn't flat. It grows with crawl size — the claim was never "84 MB no matter what," it's "bounded and predictable," which is a different and checkable promise. Growth comes from two places: redb's own page cache (tunable, default 1 GiB) and our in-RAM dedup structures — a visited-set of hashes plus a first-seen map, roughly 150 bytes per page, so ~150 MB at 1M URLs. Page content never sits in that set, only compact hashes.
- The cache is a knob, not a leak. Capping redb's cache to 96 MB dropped the 1M-page run's peak from 746 MB to 264 MB, with no change in correctness or crawl completeness — it just meant more reads hit disk instead of cache. That's the difference between "memory grows because we forgot to bound something" and "memory grows because a tunable cache is doing its job."
There's also a second storage mode: an optional in-RAM store for people who'd rather trade memory for speed and skip the disk file entirely — roughly 2.4× the RAM for about 2× the throughput, no .redb file left behind. The disk-backed store is the default because it's the one that scales past what your machine's RAM can hold; the in-RAM mode is there for smaller, throwaway crawls where you want the extra speed and don't care about resuming later.
Why this is the whole point
None of this is exotic engineering — bounded queues, streaming parsers, and transactional writes are decades-old techniques. What's unusual is applying all three to a desktop SEO crawler instead of treating "just buy more RAM" as an acceptable answer. The payoff shows up exactly when you need it least to fail: the 300k-page client site, the migration audit at 2am, the crawl you kicked off before a meeting and want to still be running when you get back.
If you want to see the gauge for yourself, point the free build at whatever site currently makes your crawler struggle and watch what happens to memory as the count climbs.