The Robots.txt Mistakes That Accidentally Deindex Entire Sites

Robots.txt is one of the smallest files on a site and one of the easiest to get catastrophically wrong, because a single misplaced character can block Google from the entire domain. The classic mistake: a staging site's robots.txt containing "Disallow: /" gets copied to production during a deploy and never removed. This single line blocks every crawler from every page on the site. It's the single most common cause of a sudden, total organic traffic collapse we see in audits, and it's almost always caught within days if anyone's actually monitoring Search Console — but sites that don't check Search Console regularly can run with this for months. A subtler version: disallowing a directory that seems safe to block — like /wp-content/ or /assets/ — without realizing that images or CSS files critical to how Google renders the page live in that directory. Google needs to render the page to evaluate mobile-friendliness and Core Web Vitals; blocking its own rendering resources can quietly hurt rankings even though the page itself isn't disallowed. Another frequent error: using Disallow rules intended to save crawl budget on faceted navigation (filtered category pages, for example) but writing the pattern too broadly, accidentally catching legitimate category pages that share a similar URL structure. A rule like "Disallow: /shop/*filter" is precise; "Disallow: /shop/*" is not, and will silently deindex the entire shop section over time as Google respects the directive and stops crawling those URLs. A five-minute audit: open robots.txt directly in a browser, read every Disallow line individually, and for each one ask "does this block anything I actually want indexed?" Then cross-reference against Search Console's Coverage report for any "Blocked by robots.txt" entries you didn't expect — that report will surface accidental blocks faster than manually reading the file, since it shows you exactly which real URLs are being caught by each rule.