How robots.txt works

Every well-behaved crawler checks /robots.txt before requesting anything else on a domain. The file is organized into groups, each starting with a User-agent line naming which crawler the rules below it apply to, followed by Disallow and Allow lines naming path patterns. Disallow: /admin/ tells that crawler not to request anything under /admin/; an Allow line can carve out an exception inside a disallowed path. A non-standard but widely honored Crawl-delay directive asks a crawler to wait a given number of seconds between requests — useful for a server that can't handle aggressive concurrency, though major engines increasingly prefer their own adaptive rate signals over a fixed number.

Crawling, not indexing — the distinction that causes the most damage

Disallowing a URL in robots.txt stops a crawler from requesting it. It does not stop that URL from appearing in search results if other pages link to it — a search engine can index a URL it's never fetched, showing it in results with no title or description, just the bare link. If the goal is genuinely keeping a page out of the index, the correct tool is a noindex meta tag or header, which requires the page to be crawlable so the crawler can see the noindex instruction in the first place. Disallowing a page in robots.txt while also relying on it to keep that page out of search results is a contradiction that produces exactly the bare-link result nobody wants.

The mistake that's easy to ship by accident

Blocking entire directories like /assets/ or /static/ to "keep crawlers out of technical files" often takes CSS and JavaScript down with it. Search engines render pages to evaluate content and layout, and a page that can't load its own stylesheet or scripts can render as broken or empty — actively hurting the pages that robots.txt was meant to help by leaving the rest of the site alone. Disallow rules should target genuinely low-value paths (internal search results, admin panels, staging duplicates), never entire static-asset directories wholesale.

Because a single misplaced Disallow can silently affect thousands of URLs, and the effect is invisible until a crawl or a rendering diff surfaces it, testing changes against real URLs before they ship matters more than reading the file by eye. RacingRoach's Robots Tester reports the exact matching directive and line for any URL and user-agent combination — see robots and sitemap validation — so you can confirm what a rule actually does before it goes live, not after crawl budget (see crawl budget) or rankings show the damage.