Robots.txt Tester

RFC 9309 defines robots.txt — a file where a site states rules for crawlers that choose to follow them.

Signal, not control: robots.txt is not access control: by itself it does not block traffic, does not guarantee exclusion and does not remove anything. This checker consults and reports — it reads what the origin says about our agent (BTOVEN-Robots) and shows it, without ever blocking the check you asked for.

What this tool does:

• Fetches /robots.txt once per origin and caches it for days — repeated checks (and every other BTOVEN checker) reuse the same result
• Parses the rule groups (User-agent / Allow / Disallow / Crawl-delay / Sitemap)
• Reports which rules apply to our agent and evaluates sample paths against them

Per RFC 9309 §2.2.1 crawlers must support robots.txt files of at least 500 KiB; an absent file means no restrictions.