How to Block AI Crawlers in robots.txt (GPTBot & Claude Guide 2026)

By /

AI crawlers arrived faster than most robots.txt files were updated. GPTBot, ClaudeBot, PerplexityBot and CCBot all obey the robots exclusion protocol, but only if you name them — a wildcard block will not stop a crawler that looks for its own user agent first. This guide lists the current agents and the rules that actually apply to them.

Interactive Client-Side Tool
Robots.txt Tester & AI Crawler Blocker Generator
● Live 2026 Engine

Generate and validate robots.txt directives to allow or block AI scraping bots (GPTBot, CCBot, Google-Extended, Bytespider, ClaudeBot).

block ai crawlers in robots.txt Complete Architecture and Verification Workflow

Table of Contents

Comprehensive Guide to block ai crawlers in robots.txt in 2026

Try Our Interactive Tool

Audit, calculate, and test in real time with our client-side utility.

Launch Free Robots.txt Tester & AI Blocker →

Explore all 18+ tools on the SBS ToolBench homepage.

Knowing how to block ai crawlers in robots.txt has become an urgent operational requirement for webmasters, digital publishers, and enterprise brands.

In 2026, over 40% of all web traffic is non-human automated bot activity. Leading Artificial Intelligence developers—including OpenAI, Anthropic, Google, ByteDance, Perplexity, and Common Crawl—routinely deploy automated spiders that scrape millions of web pages daily to train next-generation Large Language Models (LLMs) like GPT-5, Claude 3.5 Sonnet, and Gemini Ultra.

For publishers, e-commerce stores, and enterprise website owners, this creates two urgent dilemmas:

  1. Intellectual Property Theft: Your proprietary editorial research, data tables, and digital products are ingested into LLM training datasets without your consent, attribution, or financial compensation.
  2. Server CPU & Bandwidth Exhaustion: Hyper-aggressive bots like ByteDance’s Bytespider can bombard small to medium business websites with hundreds of simultaneous HTTP requests per minute, exhausting server memory, overloading Apache/Nginx workers, and causing 504 Gateway Timeout errors for real human visitors.

    3. In this practical technical guide, you will learn how to block AI training bots via robots.txt, how to differentiate between AI Training Scrapers vs. AI Search Retrieval Agents, and how to protect your server without harming your Google Search or Bing SEO rankings.

    Quick Answer (AEO Summary): The Master robots.txt Blocklist for AI Bots

    To block the most aggressive AI training scrapers from indexing your entire website, paste the following block at the bottom of your root https://example.com/robots.txt file:

    `text

    =======================================================

    BLOCK AI SCRAPERS & LLM TRAINING BOTS (2026 Master List)

    =======================================================

    User-agent: GPTBot

    Disallow: /

    User-agent: ClaudeBot

    Disallow: /

    User-agent: Anthropic-AI

    Disallow: /

    User-agent: CCBot

    Disallow: /

    User-agent: Google-Extended

    Disallow: /

    User-agent: Bytespider

    Disallow: /

    User-agent: Applebot-Extended

    Disallow: /

    User-agent: Diffbot

    Disallow: /

    User-agent: Amazonbot

    Disallow: /

    `

    Crucial Warning: Never add a blanket Disallow: / under User-agent: * unless you intend to de-index your website completely. The universal wildcard will instruct Googlebot, Bingbot, and Yahoo to delete your site from search results.

    Table of Contents

    AI Training Bots vs. AI Search Engines (The Critical Difference)

    Before adding every AI user-agent to your disallow list, you must understand the distinction between Model Training Bots and Search Retrieval Bots:

    1. Model Training Bots (Safe to Block)

    • Examples: GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider.
    • Purpose: These bots download your content offline to train foundational AI neural networks.
    • Traffic Value: Zero direct traffic. LLM training produces no referral links, citations, or clicks back to your website.
    • Recommendation: Block these if you wish to protect your intellectual property and save server bandwidth.

    2. Live Search & Retrieval Bots (Proceed with Caution)

    • Examples: ChatGPT-User, PerplexityBot.
    • Purpose: These bots crawl your web pages dynamically when a user asks a real-time question in ChatGPT Search or Perplexity AI.
    • Traffic Value: High referral traffic. When these bots read your content, they display a clickable source citation card with your brand name, sending targeted referral visitors directly to your website.
    • Recommendation: Allow these if you want your content featured in Generative Engine Optimization (GEO) answers.

    Directory of Major AI User-Agents and Their Owners

    Here is a verified breakdown of the primary AI user-agents active in 2026:

    User-Agent Name Operating Organization Primary Function Safe to Block Without Hurting SEO?
    GPTBot OpenAI Scrapes text to train ChatGPT & future GPT models Yes (Does not affect standard search indexing)
    ChatGPT-User OpenAI Live web browsing triggered by ChatGPT Search users Caution (Blocking eliminates ChatGPT referral clicks)
    ClaudeBot Anthropic Scrapes web data for Claude 3 / 3.5 model training Yes
    Anthropic-AI Anthropic Legacy / secondary crawler for Anthropic AI systems Yes
    Google-Extended Google Trains Gemini and Vertex AI models Yes (Officially separated from Googlebot)
    CCBot Common Crawl Open-source web crawl used by Meta LLaMA, Mistral & others Yes
    Bytespider ByteDance (TikTok) Aggressive crawler training Doubao and TikTok AI Yes (Frequently causes server CPU spikes)
    Applebot-Extended Apple Trains Apple Intelligence foundation models Yes (Does not affect Siri search suggestions)
    PerplexityBot Perplexity AI Real-time answer retrieval for Perplexity Search Caution (Blocks Perplexity citations)
    Diffbot Diffbot Commercial Knowledge Graph and AI web scraper Yes
    Amazonbot Amazon Trains Alexa AI and AWS Bedrock foundation models Yes

    Step-by-Step: Adding AI Disallow Rules to robots.txt

    The robots.txt file operates according to the official IETF RFC 9309 Robotics Exclusion Protocol standard. Follow these steps to implement your rules cleanly:

    Step 1: Access Your Website’s Root Directory

    Connect to your web hosting server via SFTP, SSH, cPanel File Manager, or your WordPress SEO plugin (such as Rank Math or Yoast SEO):

    • Rank Math: Navigate to WordPress Dashboard > Rank Math > General Settings > Edit robots.txt.
    • Yoast: Navigate to Yoast SEO > Tools > File Editor.

    Step 2: Preserve Existing Search Engine Directives

    Ensure your existing rules for Googlebot and Bingbot, as well as your XML sitemap link, remain completely untouched:

    `text

    User-agent: *

    Allow: /

    Disallow: /wp-admin/

    Disallow: /checkout/

    Sitemap: https://example.com/sitemap_index.xml

    `

    Step 3: Append the AI Exclusion Block

    Add your specific disallow rules at the bottom of the file. Each AI crawler requires its own dedicated User-agent: declaration followed by Disallow: /:

    `text

    Block OpenAI LLM Training

    User-agent: GPTBot

    Disallow: /

    Block Anthropic Claude Training

    User-agent: ClaudeBot

    Disallow: /

    User-agent: Anthropic-AI

    Disallow: /

    Block Common Crawl (Used by Meta LLaMA & Mistral)

    User-agent: CCBot

    Disallow: /

    Block ByteDance Spider

    User-agent: Bytespider

    Disallow: /

    `

    If your server handles ad inventory or display monetization, remember to also verify your Authorized Digital Sellers list using our Guide to Fixing ads.txt Errors in Google AdSense.

    How to Block Aggressive Scrapers at the Cloudflare WAF Level

    While reputable AI developers (OpenAI, Anthropic, Google) respect robots.txt directives within 24 hours, shady third-party scrapers and rogue botnets often completely ignore robots.txt.

    If your server CPU is spiking due to rogue scraping, block them at the network edge using Cloudflare:

    Option A: Cloudflare 1-Click AI Crawler Blocker

    1. Log into your Cloudflare Dashboard.
    2. Select your domain and navigate to Security > Bots.
    3. Toggle the switch for “Block AI Scrapers and Crawlers” to Enabled.
    4. Cloudflare will automatically block verified AI bots and scrapers across all IP ranges.

      5. Option B: Custom Cloudflare WAF Rule

      If you want to allow Perplexity or ChatGPT live search while blocking aggressive scrapers like Bytespider:

      1. Navigate to Security > WAF > Custom Rules.
      2. Click Create Rule and name it Block Malicious AI Crawlers.
      3. Set the expression:

        4. `text

        (http.user_agent contains “Bytespider”) or (http.user_agent contains “CCBot”) or (http.user_agent contains “Diffbot”)

        `

        1. Set Action to Block or Managed Challenge.
        2. Click Deploy.

          3. Google-Extended: Opting Out of Gemini Without Losing Search Rankings

          When Google announced Google-Extended, many site owners hesitated, fearing that blocking Google’s AI crawler would drop their organic rankings on Google.com.

          Google Search Central has formally confirmed: Google-Extended has zero impact on your Google Search visibility.

          `text

          User-agent: Google-Extended

          Disallow: /

          `

          • What it does: Prevents Google from using your text to train its Gemini foundation models, Imagen image generation models, and Vertex AI enterprise APIs.
          • What it does NOT do: It does not block Googlebot. Your pages will continue to be crawled, indexed, and ranked in organic Google Search, Google News, and Discover.

          Verifying Your robots.txt File for Syntax Errors

          After saving your file, verify that it is live and error-free:

          1. Direct HTTP Test: Open your browser and visit https://example.com/robots.txt. Ensure the page returns HTTP Status 200 OK and displays plain text.
          2. Google Search Console Robots Testing: In Google Search Console, open the Robots.txt Report under Settings to confirm Google has fetched the latest version.
          3. Audit Your Full Architecture: Run your domain through a comprehensive Technical Website Audit at SBS Digitals to ensure no accidental canonical or disallow conflicts are choking your organic crawl budget.

            4. Frequently Asked Questions (FAQ)

            Will blocking GPTBot hurt my Google Search rankings?

            No. GPTBot is operated exclusively by OpenAI for training ChatGPT models. It is completely independent of Google. Blocking GPTBot will have zero effect on your Google Search visibility, Googlebot indexing, or Quality Score.

            Does robots.txt legally prevent AI companies from training on my data?

            Technically, robots.txt is an advisory voluntary protocol (RFC 9309). Ethical AI companies like OpenAI, Anthropic, Google, and Apple honor it by design. However, malicious scrapers can ignore it. For complete legal protection, incorporate explicit anti-scraping language into your website’s Terms of Service and enable edge WAF rate-limiting.

            Can I allow ChatGPT to cite my website while blocking it from training?

            Yes. Allow User-agent: ChatGPT-User while adding Disallow: / for User-agent: GPTBot. This allows live ChatGPT Search to browse your pages for citations when answering user prompts, while prohibiting OpenAI’s offline bots from scraping your site for model training.

            What is Bytespider and why is it crashing my server?

            Bytespider is an automated crawler owned by ByteDance (the parent company of TikTok). Unlike Googlebot, which throttles requests based on server response times, Bytespider frequently fires thousands of concurrent requests across unoptimized database pages. Blocking Bytespider in robots.txt or via Cloudflare is highly recommended for server stability.

            What is the difference between Googlebot and Google-Extended?

            Googlebot is the web crawler responsible for crawling web pages for Google Search indexing. Google-Extended is a standalone crawler token designed solely for training Google’s generative AI models (Gemini, Vertex AI). Blocking Google-Extended will NOT remove your site from Google Search.

            Authoritative Resources & Documentation

            Author & Editorial Review

            Written by: Qasim Awan

            Lead Technical SEO Specialist & Systems Architect at SBS Digitals. Specializing in crawl budget optimization, server-side performance engineering, and bot management for enterprises across North America and Europe.

            Experiencing high server loads, bot attacks, or organic indexing issues? Book a Technical SEO Consultation with SBS Digitals today.

            [START OF SCHEMA MARKUP]

            `html

            `

            [END OF SCHEMA MARKUP]

            Need Expert Digital Marketing, SEO or Ads Management?

            Accelerate your organic rankings, lower acquisition costs, and scale your brand with proven enterprise marketing services from SBS Digitals.

            Explore SBS Digitals Services →

            For official standards and documentation, refer to the IETF RFC 9309 Robots Exclusion Protocol.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top