BlogCrawlability

Crawlability

How to read a 404 report without panicking

A crawl on a large site can turn up hundreds or thousands of 404s in one pass. Almost nobody has time to fix all of them, and almost none of them need fixing today. Here's how to tell which ones do.

A flat 404 export, versus the same URLs ranked by who actually links to them.

We've watched people open a crawl export with a few hundred 404 rows and physically flinch. It reads like a few hundred emergencies. It isn't. Most of those rows are leftover addresses nobody is currently pointing at — old bookmarks, a scraper probing for WordPress admin paths, a link from a blog that shut down in 2019. Treating every row as equally urgent is exactly what makes 404 reports feel unmanageable: you either burn a day chasing 340 dead ends, or you close the export and fix none of them.

There is a real triage question hiding in that list, and it has nothing to do with the status code itself — a 404 is a 404 whether one person will ever hit it or ten thousand will. The question that actually separates urgent from ignorable is: does something on your own site still link to this URL?

The signal that matters: who's linking to it

A 404 with no inbound link from anywhere is invisible until someone happens to type or paste that exact URL. A 404 that your own navigation, a blog post, or a sitemap still links to is different in kind — it's a broken experience you are actively serving to every user and every crawler that follows that link, and it's entirely inside your control to fix. Same status code, completely different priority.

This is what an internal link graph is for. RacingRoach's crawl records, for every URL, which pages on your own site link to it — inlink counts, not just the fact that a 404 exists. Cross-reference that against the 404 list and the undifferentiated pile splits into three buckets on its own.

The three buckets

Every 404 in a crawl falls into one of these. Work them in this order.

404 triage
Internal-linkedA page on your own site links to this URL. Fully in your control, and it's degrading every visit and crawl through that link right now. Fix the link, or 301 the URL to its correct destination — do this first.
External-onlyNo internal link, but other sites still link in — old backlinks, a press mention, a partner's outdated page. Nothing on your site is broken; you're just declining incoming link equity and a bit of referral traffic. A 301 to the closest live equivalent is polite and worth doing eventually — it's not urgent.
NoiseNo internal link, no meaningful external link — a stale bookmark, a scraper guessing paths, a tracking-parameter variant of a URL that's fine without the parameters. Safe to ignore. Chasing these is the fastest way to waste a triage session.
Same status code, three different urgencies — the internal link graph is what tells them apart.

If nothing on your own site points at a 404, it isn't costing you anything today. Spend your fix-it time where your own links are the ones going nowhere.

What this looks like on a real crawl

Here's an illustrative split — not a real measurement, just the shape a mid-size site's crawl typically takes once you run its 404 list through the same internal-link check:

347Total 404s
22Internal-linked
89External-only
236Noise
Internal-linked — fix this week22
External-only — 301 when convenient89
Noise — no action236
Illustrative only — the ratio depends on site age and how much has been restructured, but the shape (a small urgent slice, a larger ignorable one) is typical.

347 rows in an export looks like a backlog. 22 rows with your own internal links attached looks like an afternoon. That's the entire point of triage: the count doesn't change, but the amount of it you owe your attention does.

Working the list in RacingRoach

The Issues screen is where this plays out in practice — 404s show up alongside their internal inlink count, so you're sorting by "who links here" rather than scanning URLs one at a time trying to remember whether you've seen this path in your own nav. Sort internal-linked 404s to the top, fix the source link or add a 301 to wherever that content moved, and stop there for the day. External-only 404s are worth a batch pass when you have spare time — a redirect to the closest live page captures the referral traffic and link equity, but nothing breaks if it waits. Noise doesn't need a task at all; it needs you to notice it's noise and move on.

One caution: don't let "no internal link right now" become permanent amnesia. If a 404 is picking up meaningful external links or repeated hits in your server logs, it's worth a 301 even without an internal source — but that's a judgment call for the external-only bucket, not a reason to treat all 347 rows the same on day one.

RR

RacingRoach team

Technical SEO & engineering

Notes on the parts of technical SEO that don't fit in a tweet — crawl budget, rendering, and the unglamorous art of fixing redirects at scale.