LearnCrawl JavaScript sites

Guides

How to crawl JavaScript sites

A React, Vue, or Angular single-page app can ship a nearly empty HTML shell — here's how to turn on real browser rendering in RacingRoach, confirm it changed what you found, and scope it so it doesn't slow down the rest of your crawl.

The problem

A client-side-rendered single-page app (SPA) — a React, Vue, or Angular build that constructs the page in the browser after load — ships an initial HTML response that's little more than a bare <div id="root"></div> and a script tag. A text-only crawl reads exactly that response and stops: no real title, no body text, no internal <a href> links to follow. The crawl doesn't error out — it just quietly reports a page with nothing on it, which is the content and links a search engine (and your users) would actually never see either, since search engines that don't render JavaScript hit the same empty shell.

Before you reach for a rendering setting, check whether you actually need one — most sites, including most "modern framework" sites, already ship their content in the raw HTML. We cover the exact view-source / inspect-element check and the signals that tell you which camp your site is in in JavaScript rendering: when you actually need it. This guide picks up from "yes, I need it" and covers the mechanics.

Do it in RacingRoach

Rendering is a per-crawl setting, not a permanent mode — enable it on the crawl configuration screen before you start a crawl of the site or section that needs it. Once it's on, every fetch for that crawl runs through RacingRoach's headless-browser backend instead of a plain HTTP request.

That distinction matters more than it sounds: RacingRoach's fetch layer is enum-dispatched between backends, and the rendering option is a real headless browser (Chromium, driven over CDP — the Chrome DevTools Protocol) bundled with the desktop app by default, not a partial JavaScript shim that only handles simple cases. It loads the page, executes its scripts, waits for the DOM to settle, and hands off the resulting page — the same content a visitor's browser would build, not an approximation of it.

raw HTML crawl vs. rendered crawl
BackendPlain HTTP fetch, streamed straight into the parser
SeesExactly the bytes the server returned — nothing scripts would add
CostFast, low resource use — the default for a reason
BackendReal headless browser (Chromium via CDP), bundled by default
SeesThe DOM after scripts run — what a real browser would render
CostSlower and more resource-intensive per page — opt in per crawl
Same crawl engine, same issue checks — the only thing the rendering setting changes is which fetch backend produces the HTML the parser sees.

Interpret the results

Turning rendering on isn't something you should have to take on faith. RacingRoach can compare rendered vs. raw HTML per page, which is the concrete way to confirm the setting actually did something for a given URL rather than guessing from the crawl summary. Look at three things when you diff the two versions of a page:

If the rendered and raw versions of a page come back functionally identical, that's useful information too — it means this particular template didn't need rendering after all, and you can safely leave it crawling text-only.

Fix it — scope it, don't default it on

The honest tradeoff, stated plainly: rendering is slower and more resource-intensive per page than a plain HTTP fetch, because it has to load a real browser, run every script, and wait for the page to settle before handing off a single URL's worth of HTML. Turn it on for an entire large site when only the product catalog or search results are client-rendered, and you'll pay that per-page cost across pages that never needed it — for no extra data.

The practical guidance follows directly: scope rendering to the site or section that actually needs it, rather than enabling it as a blanket default. Run the bulk of a mixed site as a fast, text-only crawl, and point a separate, rendered crawl at just the client-rendered templates you confirmed above. For the full cost breakdown and the decision framework behind "should I turn this on at all," see JavaScript rendering: when you actually need it.