robots.txt Tester

about this tool

Paste a robots.txt, paste the URLs you are unsure about, and get a straight answer for each one: allowed or blocked, for the crawler you picked, and which rule on which line decided it.

The file does not work the way it reads

A robots.txt looks like a list of rules applied top to bottom. It is not, and three things about it catch people out — usually in this order.

Only one group applies. A crawler finds the group whose User-agent is the most specific match for its own name, reads that group, and ignores every other group in the file. Including the * one. So this:

User-agent: *
Disallow: /admin/

User-agent: Googlebot
Disallow: /search

does not keep Googlebot out of /admin/. The moment a Googlebot group exists, Googlebot stops reading the * rules entirely. Everything you wanted to keep has to be repeated inside the new group. This is the single most common way a robots.txt does the opposite of what was intended.

Order decides nothing. The rule with the longest path pattern wins, wherever it sits in the file. Moving a line up changes nothing at all.

Allow breaks a tie. When two matching patterns are exactly the same length, Allow wins. Allow: /abc against Disallow: /ab* is that case. Note that Allow: /folder/ beating Disallow: /folder is not — eight characters against seven, so the longer one wins before the tie-break is reached.

Nothing is fetched

You paste the file, or drop it in. This does not go and get it.

That is deliberate twice over. A tester that fetches your robots.txt tells you about the copy a datacentre is served, which is not always what your users get through a CDN or an edge rule. And it cannot test a file you have not published yet — which is the case you most want to check, because the point of testing is to find out before it goes live rather than after.

What the patterns mean

Every pattern is a prefix unless it ends in $. * stands for any run of characters, including none. Everything else is literal, and paths are case-sensitive. Both the rule and the URL are percent-encoded before they are compared, as RFC 9309 requires, so /café and /caf%C3%A9 are the same path whichever way you type it.

| Pattern | Matches | Does not match | |---|---|---| | /fish | /fish, /fish.html, /fishheads | /Fish.asp, /catfish | | /fish/ | /fish/, /fish/salmon.htm | /fish, /fish.html | | /*.php | /index.php, /filename.php?parameters, /filename.php5 | /, /windows.PHP | | /*.php$ | /filename.php | /filename.php?parameters, /filename.php5 |

Those rows are Google's own examples, and the tests for this tool check against them rather than against its own output — the only way to know a matcher agrees with the crawler it is imitating.

Blocked is not the same as hidden

Disallow stops a page being crawled. It does not stop it being indexed. A URL that other sites link to can still turn up in results, listed without a description, because the crawler knows the address exists but was never allowed to read the page.

To keep something out of the index you need a noindex meta tag or an X-Robots-Tag header — and the crawler has to be allowed to fetch the page to see either of them. Blocking it in robots.txt guarantees it never will.

Noindex: written inside robots.txt has never been part of the standard, and Google stopped honouring it in 2019. It is flagged here as a problem, because a file carrying one usually belongs to someone who thinks a page is hidden.

What it checks beyond the verdict

Rules written before any User-agent line, which no crawler reads. Rules that are full URLs rather than paths. Sitemap: lines that are paths instead of absolute URLs — and if you want to know whether the file at the other end of one is valid, the sitemap validator reads it. A byte order mark at the top of the file. And Crawl-delay, which is a note rather than a fault — Bing still honours it, Google never has and Yandex stopped in 2018.

A misspelled directive is the interesting one. Google's own parser quietly corrects Disalow, Dissallow and several other spellings, and reads a line whose colon is missing. So those lines are obeyed here too — a tester that called them unreadable would print the opposite verdict to the crawler — and flagged, because most other crawlers are stricter and the rule ends up working for Google alone. Anything that is not a directive at all gets its own panel rather than being swallowed.

The same crawler named in two groups is not an error, and this says so: RFC 9309 requires a crawler to combine them, so nothing in either group is lost. It is listed as untidy, not broken.

Past 500 KiB the file is cut, because that is where a crawler stops reading. Answering from a rule on line 30,000 that no crawler ever saw would be worse than not warning at all.

If the URL you are testing is reached through a redirect, the redirect tester shows the chain it travels first — a rule can be allowed here and never reached in practice.

For the other half of what a crawler sees, the meta tag generator writes the noindex and canonical tags that decide indexing rather than crawling, and the HTTP header analyser reads the X-Robots-Tag a server can send instead. To pick a single URL apart before testing it, the URL parser.

questions

Why does a rule under User-agent: * not apply to Googlebot?

Because a crawler reads one group. It finds the group naming its own product token and ignores every group naming a different crawler, including the * one. So the moment you add a Googlebot group, Googlebot stops reading the * rules entirely — and anything you wanted to keep has to be repeated inside the new group. Groups that name the same crawler twice are the exception: RFC 9309 says those are combined, so nothing in them is lost.

Two rules match my URL. Which one wins?

The one with the longer path pattern, wherever it sits in the file. Order decides nothing. Allow: /folder/ beats Disallow: /folder because it is eight characters against seven, not because it is an Allow. The Allow rule only wins when the two patterns are exactly the same length — Allow: /abc against Disallow: /ab* is the tie that needs it.

Does this fetch my robots.txt?

No. You paste the file or drop it in, and the matching runs in the page. A tester that fetches the file tells you about the copy a datacentre is served, which is not always the one your users and crawlers get — and it cannot test a file you have not published yet, which is the case you most want to check.

Does Disallow with nothing after it block everything?

No, the opposite — RFC 9309 says a rule with an empty path is ignored, so Disallow: on its own is the usual way of writing "nothing is blocked here". Disallow: / with the slash is the one that blocks the site.

What do * and $ mean in a path?

* stands for any run of characters, including none, and $ anchors the pattern to the end of the URL. Without a $ every pattern is a prefix, which is why /fish also matches /fishheads. Paths are case-sensitive: /Fish.asp does not match /fish.

Can I stop a page appearing in search with robots.txt?

Not reliably. Disallow stops the page being crawled, not indexed — a URL that other sites link to can still appear, listed without a description. To keep it out of the index you need a noindex meta tag or an X-Robots-Tag header, which the crawler can only see if the page is not blocked. Noindex inside robots.txt has never been part of the standard and Google stopped honouring it in 2019.

Why is my Crawl-delay ignored?

Google has never supported it, and Yandex stopped in February 2018. Bing still reads it. It is flagged here as a note rather than an error because it does still work for one major crawler — it just will not do what most people add it for.

Does it handle a misspelled directive?

Yes, the same way Google does. Google’s parser silently corrects Disalow, Dissallow and a few other spellings, and accepts a line whose colon is missing entirely — so a tester that called those unreadable would print the opposite verdict to the crawler. They are obeyed here and flagged, because most other crawlers are stricter and the rule will work for Google and nobody else.

Why does Googlebot-Image sometimes read the Googlebot rules?

Because Google documents that its image, news and video crawlers fall back to the Googlebot group when nothing names them specifically. AdsBot is the opposite exception: it never reads the * group at all, so a site blocked only by User-agent: * is still crawled by it. Both are modelled here.