Reachability · document 01

Robots Policy Simulator

In the workspace: /robots-policy

Availability

Included on Basic, Pro and Advanced

This document is open to every reader on every plan. Availability tells you what a plan includes; it does not decide what you may read.

What it is for

A robots.txt file is a set of instructions, not a lock. It tells a crawler which paths it may request; it does not stop anyone from requesting them. This instrument takes the rules your site serves right now, applies them to one address for one named crawler, and reports the decision those rules produce together with the exact line that produced it. Use it before you publish a rule, and after: the question "did I just block my own product pages" has a checkable answer.

How to use it

Choose the project whose robots.txt you want to test. The instrument reads the file your site is serving at that moment, not a copy you paste in.
Choose the crawler to simulate. Googlebot and a general agent behave differently the moment your file names one of them specifically.
Enter the path you are asking about. A full URL or a path both work; only the path is compared.
Read the verdict, then read the line beneath it. The line is the part worth keeping.

Reading the result

The verdict is Allow or Block for that one agent against that one path, and it names the matching directive and the layer it came from. When two rules could both apply, the more specific rule is shown as the deciding one rather than left for you to work out. A Block verdict on a path you meant to publish is the finding; a Block verdict on an admin path is the rule working.

What it cannot tell you

It proves what your rule set says. It cannot prove what a crawler will do, because a directive is a request: honoured by the major engines, ignored by anything that chooses to. And a path excluded here can still appear in search results when another site links to it, because exclusion from crawling is not exclusion from indexing. If a page must stay out of an index, a robots rule is the wrong instrument for the job.

What it keeps

Nothing. The address you test and the verdict you receive live in this browser session and are cleared when you sign out. No row is written to the database.