How it works
Built to stay fast when the crawl gets huge.
Most desktop crawlers slow to a halt on large sites because they keep the whole crawl in memory. RacingRoach is built the other way around: a bounded, streaming engine that writes results to disk as it goes, so RAM stays in the hundreds of megabytes even at a million URLs — instead of the tens of gigabytes a RAM-bound crawler needs.
Bounded & backpressured
The URL frontier and fetch pipeline are bounded end to end. When storage is the bottleneck, crawling slows down — it never balloons memory to keep up.
Streamed to disk
By default, every page record, link, and issue commits to an embedded on-disk database as it completes, in crash-safe transactions — the full crawl never lives in RAM. Prefer speed over a disk file? An optional in-RAM storage mode keeps the store in memory instead.
Resumable
Each crawl is one reopenable file. Close the app, lose power, come back tomorrow — reopen it and keep going, exactly where you left off.
The invariant
Never hold the full crawl in RAM.
That single rule is enforced through the whole engine — frontier, fetch, parse, store. It isn't a setting you switch on for big sites; it's how the tool is built, which is why RAM stays bounded — hundreds of MB, not tens of GB — and why RacingRoach keeps its speed where RAM-bound crawlers stall.
Private by construction
Your crawl is yours.
There's no cloud service behind RacingRoach — nothing to upload, no account to create, no usage phoned home. A closed-by-default safety guard blocks private ranges and cloud-metadata endpoints, re-checked on every redirect hop, with bounded response bodies. When you audit a client's staging site, it stays on your machine.
See how little RAM it uses.
Point the free build at the site that gives your current crawler trouble, and watch what the memory gauge does.