BlogEngineering

Engineering

Designing a crawler that triages instead of dumps

A crawl that surfaces 8,000 issues and ranks none of them isn't a report — it's a filing problem you've handed to the user. Here's why we bucket severity into the data itself, in the engine, before it ever reaches a screen.

Same 90 checks, two different products: a flat export, and a ranked worklist.

Point most crawlers at a mid-size site and you get a spreadsheet back: one row per URL, one column per check, thousands of cells. That's not a failure of the crawl — the crawl worked fine. It's a failure of what happens to the results afterward. A 50,000-row CSV with a missing-title flag sitting in the same column as a broken-canonical flag isn't information. It's a filing problem, and you just inherited it.

We built the Issues screen to refuse that handoff. Every one of our 90+ checks (the full catalog — 404s, redirect chains, duplicate titles, missing structured data, slow response times, and dozens more) gets sorted into one of four severity buckets the moment it's detected, not after. That sounds like a small ordering choice. It's actually the central design decision behind the whole screen, and it has consequences that reach all the way down into where the classification code lives.

The default failure mode: completeness without triage

Most crawler output optimizes for one thing: don't miss anything. That's the right instinct as far as it goes — an audit tool that silently drops issues is worse than useless, it's actively misleading. But "don't miss anything" and "tell me what to do first" are different problems, and a lot of tools solve only the first one.

The result is a table where a page returning a 500 and a page with a title three characters over the recommended length are visually identical: one row, one flag, equal weight. Nothing in the export tells you which one is costing you indexed pages right now and which one is a nice-to-have for next sprint. You either already know the technical-SEO priority order from experience, or you re-derive it by hand, every time, for every site. That's real, repeated cognitive work the tool should have done once.

A fixed taxonomy, not a filter

Our answer is a closed set of four severities, and every check in the catalog belongs to exactly one:

CriticalBroken internal links, server errors (5xx), redirect chains, missing page titles — the page is unreachable, gone, or invisible to search.
WarningDuplicate titles, missing meta descriptions, multiple H1 tags — degrades ranking or display, nothing is actually broken.
OpportunitySlow response time, large page weight, missing canonical — the page works, it could work better.
Noticenoindex, nofollow, URL parameters — informational, usually intentional, rarely worth touching.

The taxonomy is deliberately small and deliberately fixed. We didn't build a rules engine where you assign your own severity per check, and we didn't leave it as an open scale (a 1–10 priority score, say) that would need constant recalibration. Four buckets, decided once, the same across every crawl and every user. That rigidity is the point — a severity score only works as a shared vocabulary if it means the same thing every time you see it.

Why the taxonomy lives in the engine, not the UI

The tempting shortcut is to keep every check's output flat and let the interface layer sort it — a "severity" column computed by a lookup table in the frontend, applied at render time. We didn't do that, and the reason is architectural, not aesthetic.

In the engine, kinds_for — the function in issues.rs that classifies a single crawled page — doesn't just decide whether a page has a missing title or a slow response. It returns IssueKind values, and every IssueKind carries a Severity as an intrinsic property, fixed by a single exhaustive match (BrokenInternalLink → Critical, DuplicateTitle → Warning, SlowResponse → Opportunity, and so on) baked into the engine, not attached later.

That placement matters because of everything downstream of classification that doesn't go through the UI at all: CSV export, saved segments, the crawl-overview dashboard's health-score rollup, cross-crawl comparison. If severity were a display-layer concern, every one of those paths would need its own copy of the same lookup table, and they'd drift the moment one of them was updated and the others weren't. Because severity is a property of the IssueKind itself, it's already correct in the CSV row, already correct in a filtered segment, already correct anywhere a PageRecord's issues get read — with nothing to keep in sync.

Severity isn't a view of the data. It's part of the data. That's what makes it survive export, filtering, and segmentation without a second implementation anywhere.

The wrinkle: not every check fits in one page's classification pass

Most checks are per-page: look at one crawled record, decide if it trips a rule, done — that's what kinds_for does, and it's stateless, one record in, a list of kinds out. But some checks are inherently cross-record. Duplicate-title detection can't be decided by looking at a single page; a title is only a duplicate in relation to every other title in the crawl. The same is true for duplicate meta descriptions, duplicate H1s, and duplicate body content.

Those checks are computed by the store instead, in a separate pass that scans the whole crawl and looks for matches across records — architecturally distinct from kinds_for, but assigned a severity through the exact same fixed Severity enum. The classification mechanism forks in two places (per-page vs. cross-record); the severity taxonomy itself doesn't. A user looking at the Issues screen never has to know or care which path produced a given row — a duplicate title and a missing title both show up ranked by the same four buckets, because the taxonomy is the contract, not an implementation detail of how each check happens to be computed.

Completeness vs. usefulness, and why we pick a side

There's a real tension here, and we don't think it fully resolves. "Show me everything you found" and "tell me what to fix first" pull in different directions: full completeness risks burying the urgent under the trivial; aggressive triage risks hiding something a user actually needed to see. A tool that only ever shows you critical issues and quietly suppresses the rest would be lying by omission.

We come down on the triage side, deliberately, but not by deleting anything — every check still runs, every result still exists, opportunities and notices are still one click away. What we refuse to do is present them with equal visual and structural weight to a broken page or a missing title. The severity-ordered checklist is the reader-facing version of this same argument: fix critical the same day, batch warnings, revisit opportunities when there's engineering time, skim notices and move on. That ordering isn't a UI convenience layered on top of a flat list. It's the shape the data was always in, going back to the moment each check ran.

RR

RacingRoach team

Technical SEO & engineering

Notes on the parts of technical SEO that don't fit in a tweet — crawl budget, rendering, and the unglamorous art of fixing redirects at scale.