Policy playground · local and unlimited
Robots playground
Test a robots.txt against one client and path. See the matching groups, the selected rule, and the reason for the decision.
Allowed for OAI-SearchBot
The named group allows /, which matches /guides/agent-access. The separate GPTBot group disallows crawling for that product token.
This is the sample above, not a live site check.
From policy text to a decision
- Select the client’s groups.Matching named groups are combined. The wildcard group applies when no specific group matches.
- Match the requested path.The longest matching Allow or Disallow rule takes priority. Allow wins an equally specific tie.
- Read it as a policy.Robots rules tell cooperating clients what you permit. They do not authenticate a bot or prove that a firewall will serve the page.
Robots Exclusion Protocol (RFC 9309) · Test delivery and indexing too