Robots.txt Tester and Validator

It does: parse your robots.txt into groups, pick the right group for the user-agent you choose, and apply the same longest-match-wins logic Google uses - including * wildcards, the $ end-anchor, and Allow winning ties. It also flags unsupported directives like Noindex and Crawl-delay, and paths that are missing a leading slash.

It does not: fetch your live robots.txt for you - you paste it in, so you can test changes before they go live. It also will not model every quirk of every crawler, redirect handling, or URL canonicalisation. Treat it as a fast, accurate check of your rules, not a replacement for Search Console once the file is deployed.

When a crawler wants to fetch a URL, it does not just scan your robots.txt top to bottom and stop at the first match. Google, and most serious crawlers, follow a specific algorithm, and this tester implements it so you get the same verdict the crawler would.

Step one is choosing a group. A robots.txt is a series of groups, each beginning with one or more User-agent lines. The crawler finds the group whose user-agent most specifically matches its own name. If your file names Googlebot explicitly, Googlebot uses that group and ignores the User-agent: * group entirely. This trips people up constantly: adding a specific bot block silently removes all your general rules for that bot. If no named group matches, the crawler falls back to the * group. If there is no * group either, everything is allowed.

Step two is choosing a rule. Within the chosen group, the crawler looks at every Allow and Disallow line and finds the one whose path most specifically matches the URL. Specificity is measured by the length of the path pattern - the longer the matching pattern, the more it wins. So Allow: /folder/page beats Disallow: /folder/ for a URL inside that folder, because the Allow pattern is longer. When an Allow and a Disallow match with equal length, Allow wins. An empty Disallow: is a rule that matches nothing, which is how you say "allow everything".

Wildcards matter. A * matches any run of characters, so Disallow: /*?add-to-cart= blocks every URL containing that query string. A $ anchors the end, so Disallow: /*.pdf$ blocks PDFs but not /file.pdf?ref=x. This tool supports both.

Two things worth repeating, because they cause real damage. First, blocking a URL here does not remove it from Google - it only stops the crawl. A blocked URL can still be indexed from external links, just without a description. To remove a page, allow it to be crawled and use a page-level noindex. Second, a stray Disallow: / in the wrong group will delist an entire site, and it is the single most common catastrophic robots.txt error. Test every change here before you deploy it, and verify again in Search Console afterwards.