racingroach

A performance-first, open-source website crawler.

Built around one hard rule: never hold the full crawl in RAM. Every data path is bounded, streaming, and backpressured — results stream to disk as they complete.