robots.txt Tester & AI Blocker Generator
Test your robots.txt against Googlebot, Bingbot, and AI LLM crawlers. Includes verified 1-click templates to block AI training bots without harming SEO.
robots.txt Editor & Presets
Live URL Access Simulator
Select a user-agent and path to test if it is permitted or blocked by the rules on the left:
Testing robots.txt Against Real Crawlers
A single misplaced slash in robots.txt can deindex a section of your site, and you often find out only when traffic drops. This tester runs your rules against Googlebot, Bingbot and the current AI crawlers, and names the directive that matched for any URL you check.
Paste your robots.txt and a URL above. The tester shows whether that URL is allowed or blocked for each user agent, and points at the exact line responsible.
🛠️ Practitioner Pro Tip: Why Crawl-Delay is Ignored by Googlebot and Scrapers
Googlebot has officially deprecated the Crawl-delay directive for years, and aggressive scraping bots (such as ByteSpider and CCBot) frequently ignore robots.txt entirely. To truly stop AI scraping bots from eating server bandwidth, combine RFC 9309 robots.txt rules with edge firewall rules in Cloudflare WAF to challenge non-verified autonomous user-agents.
How the Robots Exclusion Protocol Resolves Rules
RFC 9309 settled how conflicts resolve: the most specific matching rule wins, not the first one in the file. Allow beats Disallow when the matched path is longer, a user-agent block applies only to the agents it names, and a crawler that finds its own block ignores the wildcard block entirely.
The tester implements that precedence rather than reading the file top to bottom, supports wildcards and end-of-string anchors, and includes the current AI crawler user agents so you can see at a glance which of them your file actually blocks.
Crawler User Agents
For more detailed technical walkthroughs, explore our in-depth article on How to Block AI Crawlers in robots.txt. You can also browse our full directory of 18+ free web utilities on the SBS ToolBench homepage.
robots.txt Questions
Where does the robots.txt file belong?
The robots.txt file must always be placed in the root directory: https://example.com/robots.txt. It cannot be hosted in a subdirectory.
Does blocking GPTBot affect my Google rankings?
No. GPTBot is operated solely by OpenAI for model training and has zero impact on Google Search indexing or organic ranking.
How often do search engines recrawl robots.txt?
Major search engines cache robots.txt for up to 24 hours. Purging server cache ensures instant discovery of updated disallow directives.
📚 Official Companion Guides & Industry Blueprints
Deepen your technical knowledge with step-by-step documentation designed to accompany this utility.