Robots.txt for AI: How to Control AI Crawlers & Protect Your Content (2026)

AI crawlers are automated bots operated by companies like OpenAI, Anthropic, Google, and Perplexity that visit your website to collect content for training AI models, powering AI search features, or both. Your robots.txt file is the primary mechanism for controlling which AI crawlers can access your content — and most websites in 2026 have not configured it for the AI era at all.

Right now, dozens of AI crawlers are visiting websites across the internet, downloading content at scale, and feeding it into large language models, training datasets, and AI search engines. Some of these crawlers identify themselves honestly. Others use ambiguous user-agent strings. And your robots.txt file — a simple text file that has existed since 1994 — is the front line of defense for deciding who gets access to your content and who does not.

The problem is that most website owners do not know these crawlers exist, let alone how to manage them. The result is an uncontrolled free-for-all where AI companies harvest content with no restrictions. On the other side of the spectrum, some websites have overreacted by blocking every AI crawler — inadvertently killing their visibility in ChatGPT, Perplexity, and other AI-powered search platforms that now drive real referral traffic.

This guide is the definitive reference for managing AI crawlers with robots.txt in 2026. It includes a complete directory of every known AI crawler, copy-paste configurations for four different strategies, and a clear framework for deciding what to block and what to allow based on your specific business goals.

What Are AI Crawlers? How They Differ from Search Engine Bots

AI crawlers are web bots that download your content for AI-related purposes: training machine learning models, powering real-time AI search responses, or building retrieval-augmented generation (RAG) indexes. They differ from traditional search engine crawlers like Googlebot and Bingbot in several critical ways.

Traditional search crawlers (Googlebot, Bingbot) index your content so it appears in search results. When a user clicks a search result, they visit your website. There is a clear value exchange: you allow crawling, and you receive organic traffic in return. This model has been the foundation of the web for over 25 years.

AI training crawlers (GPTBot, CCBot, Bytespider) download your content to train AI models. Your content becomes part of the model's knowledge, but there is typically no attribution, no link back, and no traffic sent to your website. This is a one-directional extraction of value — the AI company benefits, but you may not.

AI search crawlers (ChatGPT-User, PerplexityBot, OAI-SearchBot) access your content in real-time when users ask questions. They generate AI-powered answers that cite your website as a source, often with a link. This model is closer to the traditional search value exchange — you receive traffic and attribution in return for access.

Understanding this distinction is essential because it determines your robots.txt strategy. Blocking AI training crawlers protects your intellectual property. Blocking AI search crawlers removes you from a growing traffic channel. The optimal approach for most websites is to allow one category while restricting the other.

Complete AI Crawler Directory (2026)

This is the most comprehensive AI crawler reference table available. It covers every major AI bot that may be visiting your website, what company operates it, what it does with your content, and whether it is allowed by default if you have no specific rules in your robots.txt.

AI Crawler Traffic Share (2026 Estimates)

Based on aggregated server log analyses across thousands of websites, these are the estimated traffic share percentages of the major AI crawlers in 2026:

GPTBot is by far the most active AI crawler on the internet, accounting for roughly 45% of all AI bot traffic. PerplexityBot has grown rapidly since 2024, reflecting Perplexity's surge in popularity as an AI search engine. ClaudeBot's share is smaller but growing steadily. The "Others" category includes Bytespider, CCBot, FacebookBot, cohere-ai, and other less common crawlers.

How robots.txt Works — A Quick Refresher

The robots.txt file is a plain text file placed at the root of your website (e.g., https://example.com/robots.txt) that tells web crawlers which pages or sections of your site they are allowed or not allowed to access. It has been a web standard since 1994 and is formalized as RFC 9309.

The file uses a simple syntax with three core directives:

# Basic robots.txt syntax
User-agent: BotName      # Which bot this rule applies to
Disallow: /private/       # Block access to this path
Allow: /public/           # Explicitly allow access to this path
Sitemap: https://example.com/sitemap.xml  # Tell bots where your sitemap is

Key rules to understand:

• User-agent: * applies to ALL bots. Specific User-agent rules override the wildcard for that specific bot.
• Disallow: / blocks access to the entire site for the specified bot.
• Disallow: (empty value) allows access to the entire site for the specified bot.
• More specific rules win. If you have Disallow: /blog/ and Allow: /blog/public/, the bot can access /blog/public/ but nothing else under /blog/.
• robots.txt is voluntary. Bots are asked to respect these rules, but they are not technically forced to. Legitimate companies (OpenAI, Google, Anthropic, Perplexity) honor robots.txt. Rogue scrapers may not.
• Each bot group needs its own block. You cannot combine multiple User-agent lines with a single set of rules (though some implementations support this, it is safest to separate them).

The Decision Process: How to Decide What to Block

Before writing any robots.txt rules, you need a clear decision framework. Randomly blocking or allowing bots without a strategy leads to either over-blocking (losing AI traffic) or under-blocking (giving away content for free). Use this five-step process:

Step 1: Identify which AI bots are visiting your site. Check your server access logs for user-agent strings matching the bots in the directory table above. Most websites are surprised by the volume of AI crawler traffic they receive — some sites see more AI bot requests than human visitors.

Step 2: Assess the value exchange. For each bot, ask: "Does allowing this bot benefit my website?" PerplexityBot sends referral traffic with clear citations. GPTBot takes training data with no direct benefit to you. The answer determines whether to allow or block.

Step 3: Write your configuration. Based on your assessment, choose one of the four strategies below and implement the corresponding robots.txt rules.

Step 4: Test your configuration. Use Google's robots.txt tester (in Google Search Console) and the robots.txt validation tools built into most SEO suites to verify your syntax is correct. A single typo can accidentally block all crawlers or allow ones you intended to block.

Step 5: Monitor results. After implementing your rules, track your AI referral traffic in GA4 (referrals from chatgpt.com, perplexity.ai, claude.ai) and your bot traffic in server logs. Adjust your strategy based on what you observe.

4 Strategic Approaches to AI Crawler Management

There is no single "correct" robots.txt configuration for AI crawlers. The right approach depends on your content type, business model, and strategic goals. Here are the four primary strategies, with clear guidance on when each is appropriate.

Strategy 1: Block All AI Crawlers

Best for: Paywalled content, proprietary research, premium publications, legal/medical content databases, and any business where content IS the product.

This is the most protective approach. You block every known AI crawler from accessing any part of your website. Your content will not be used for AI training, will not appear in ChatGPT or Perplexity responses, and will not be cited by any AI search engine. You are invisible to the entire AI ecosystem.

When to use it: If your revenue depends on users visiting your website to access content (subscriptions, paywalls, lead generation through gated content), blocking AI crawlers prevents that content from being summarized and served for free by AI systems. Major publishers like The New York Times and The Wall Street Journal use this approach.

The trade-off: You receive zero referral traffic from AI search platforms. As AI-powered search grows, this means an increasing share of potential visitors will never discover your content. You also lose any potential for AI citations, which are becoming a form of digital authority.

Strategy 2: Allow All AI Crawlers

Best for: Open-source projects, educational resources, government websites, non-profits, and any content whose mission is maximum distribution.

The simplest approach: do nothing. If your robots.txt has no specific AI crawler rules, all bots are allowed by default. Your content will be used for training, appear in AI search results, and be cited across platforms. This maximizes your AI visibility and potential referral traffic.

When to use it: If your goal is to spread information as widely as possible — open-source documentation, academic research, public health information, or government resources — allowing all AI crawlers ensures your content reaches the maximum possible audience, including through AI platforms.

The trade-off: Your content will be used to train AI models without compensation. AI systems may summarize your content so thoroughly that users never visit your website. You have no control over how AI systems represent your content or context.

Strategy 3: Selective Allow (Recommended for Most Businesses)

Best for: Most businesses, blogs, e-commerce sites, SaaS companies, and agencies that want AI search traffic but want to protect their content from training.

This is the strategy we recommend for the majority of websites. You block training-focused crawlers (GPTBot, CCBot, Bytespider, anthropic-ai, cohere-ai) while allowing search-focused crawlers (ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot). This way, your content appears in AI search results with attribution and referral traffic, but is not used to train competing AI models.

When to use it: If you want the benefits of AI search visibility (citations, referral traffic, authority building) without giving away your content for model training. This is the optimal balance for most content-driven businesses in 2026.

The trade-off: The distinction between "search" and "training" is not always clear-cut. Some companies may use search crawling data to improve their models indirectly. However, by blocking the explicitly training-focused crawlers, you send a clear legal and technical signal about your content use preferences.

Strategy 4: Tiered Access by Content Section

Best for: Large websites with diverse content types — e-commerce with blog and product pages, SaaS with documentation and pricing pages, publishers with free and premium content.

The most sophisticated approach: you apply different rules to different sections of your website. For example, you might allow AI crawlers to access your public blog (which benefits from AI citations) while blocking them from your product catalog (which contains proprietary pricing and descriptions), your customer support area, and your internal documentation.

When to use it: When different parts of your website have different value propositions for AI crawler access. Your blog benefits from AI citations and referral traffic. Your product data, pricing, or proprietary content does not.

The trade-off: More complex to configure and maintain. You need to ensure your URL structure is clean enough that Disallow and Allow rules can effectively target the right sections. Requires regular auditing as new pages and sections are added.

Copy-Paste robots.txt Configurations

Here are four ready-to-use robots.txt configurations, one for each strategy. Copy the configuration that matches your chosen strategy and add it to your robots.txt file. These configurations cover all known AI crawlers as of March 2026.

Configuration 1: Block All AI Crawlers

# ============================================
# BLOCK ALL AI CRAWLERS
# Prevents AI training AND AI search indexing
# ============================================

# OpenAI (ChatGPT, GPT models)
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

# Anthropic (Claude)
User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

# Google AI Training (does NOT affect Google Search)
User-agent: Google-Extended
Disallow: /

# Perplexity
User-agent: PerplexityBot
Disallow: /

# ByteDance (TikTok)
User-agent: Bytespider
Disallow: /

# Common Crawl
User-agent: CCBot
Disallow: /

# Meta (Facebook/Instagram AI)
User-agent: FacebookBot
Disallow: /

# Cohere
User-agent: cohere-ai
Disallow: /

# Apple Intelligence
User-agent: Applebot-Extended
Disallow: /

# Allow regular search engines
User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

Sitemap: https://example.com/sitemap.xml

Configuration 2: Allow All AI Crawlers

# ============================================
# ALLOW ALL AI CRAWLERS
# Maximum AI visibility and discoverability
# ============================================

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Configuration 3: Selective Allow (Recommended)

# ============================================
# SELECTIVE: Block Training, Allow Search
# Best balance for most websites (2026)
# ============================================

# BLOCK — AI Training Crawlers
User-agent: GPTBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# ALLOW — AI Search Crawlers (provides citations + traffic)
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

# ALLOW — Traditional Search Engines
User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

Sitemap: https://example.com/sitemap.xml

Configuration 4: Tiered Access by Content Section

# ============================================
# TIERED: Different rules per content section
# Blog = open, Products/API = protected
# ============================================

# Block all AI training bots entirely
User-agent: GPTBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

# AI Search bots: allow blog, block products & internal
User-agent: ChatGPT-User
Allow: /blog/
Allow: /guides/
Disallow: /products/
Disallow: /api/
Disallow: /account/
Disallow: /admin/

User-agent: OAI-SearchBot
Allow: /blog/
Allow: /guides/
Disallow: /products/
Disallow: /api/
Disallow: /account/
Disallow: /admin/

User-agent: PerplexityBot
Allow: /blog/
Allow: /guides/
Disallow: /products/
Disallow: /api/
Disallow: /account/
Disallow: /admin/

User-agent: ClaudeBot
Allow: /blog/
Allow: /guides/
Disallow: /products/
Disallow: /api/
Disallow: /account/
Disallow: /admin/

# Traditional search engines: full access
User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

Sitemap: https://example.com/sitemap.xml

Content Type Decisions: What to Block and What to Allow

Not all content has the same value proposition for AI crawler access. Use this priority grid to determine the right approach for each type of content on your website:

When making these decisions, consider the following principles:

• Content that benefits from distribution should be allowed. Blog posts, guides, how-to articles, and educational content all benefit from wider distribution through AI platforms. More citations mean more authority and more traffic.
• Content that IS the product should be protected. If users pay to access your content (subscriptions, courses, research reports), allowing AI crawlers to summarize it for free undermines your business model.
• Content with competitive value should be evaluated carefully. Product descriptions, pricing data, and proprietary methodology are competitive assets. Allowing AI training on this data could help competitors who use those same AI models.
• Private content should always be blocked. User accounts, admin panels, internal tools, and customer data should be blocked from ALL crawlers, not just AI ones. This is a basic security practice.

Beyond robots.txt: Additional Content Protection Methods

While robots.txt is the primary tool for managing AI crawlers, it is not the only one. Several other mechanisms exist for communicating your content use preferences to AI systems, and some offer stronger protections.

Meta Robots Tags

The <meta name="robots"> tag in your HTML provides page-level control over crawling and indexing behavior. While traditionally used for search engines, Google has introduced AI-specific directives:

<!-- Block Google AI training for a specific page -->
<meta name="googlebot" content="noai, noimageai">

<!-- Standard robots directives (still essential) -->
<meta name="robots" content="index, follow">

The noai directive tells Google not to use this page's content for AI training (Gemini), while noimageai specifically blocks image use. These are page-level controls, making them more granular than robots.txt rules which operate at the directory level.

X-Robots-Tag HTTP Header

For non-HTML content (PDFs, images, documents), you can use the X-Robots-Tag HTTP header to communicate the same directives:

# In .htaccess or server config
Header set X-Robots-Tag "noai, noimageai"

This is particularly useful for protecting images, PDFs, and other files that do not have an HTML <head> section where you could place a meta tag.

The ai.txt Proposal

Several industry groups have proposed ai.txt as a dedicated standard for communicating AI content use policies — separate from robots.txt. The ai.txt proposal allows website owners to specify whether their content can be used for training, whether attribution is required, and what license terms apply. As of March 2026, ai.txt is not yet a formally adopted standard, but several major AI companies have expressed support for it. It is worth monitoring.

TDM (Text and Data Mining) Policies

The EU's Digital Single Market Directive and similar legislation in other jurisdictions have established legal frameworks around text and data mining. TDM reservation headers (TDMRep) allow website owners to legally reserve their rights over content used for text and data mining, including AI training. Implementing a TDM policy is a legal complement to the technical controls provided by robots.txt.

The SEO/AEO Trade-off: What You Gain and Lose

Every robots.txt decision involves a trade-off between content protection and AI visibility. Blocking AI crawlers protects your content from being used without compensation. Allowing AI crawlers positions your website as a source that AI systems cite, recommend, and send traffic to. Understanding this trade-off quantitatively helps you make better decisions.

What you gain by allowing AI search crawlers:

• AI referral traffic: Websites that appear in Perplexity citations, ChatGPT browsing results, and Google AI Overview receive measurable referral traffic. Early data suggests AI referral traffic is growing 3-5x year over year for optimized sites.
• Brand authority: When AI systems consistently cite your website as a source, it builds brand recognition and perceived authority among the growing audience that uses AI search as their primary information tool.
• AEO/GEO scores: Allowing AI crawlers is a prerequisite for Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO). If bots cannot access your content, you cannot optimize for AI citations.
• Competitive advantage: If your competitors block AI crawlers and you do not, AI systems will cite you instead of them — potentially capturing traffic and authority that would have gone to competitors.

What you lose by allowing AI training crawlers:

• Content exclusivity: Your content becomes part of AI training datasets. AI systems may generate responses that effectively replicate your content without attribution, reducing the unique value of visiting your website.
• Competitive risk: Competitors who use AI tools trained on your content indirectly benefit from your work. Your proprietary methodology, unique data, and creative output become part of a shared model.
• Bandwidth costs: AI crawlers can be aggressive, consuming significant server bandwidth. GPTBot in particular has been reported to make thousands of requests per day to individual websites, which can impact server performance and increase hosting costs.

For most businesses, the strategic sweet spot is the Selective Allow approach: block training bots to protect your intellectual property while allowing search bots to gain the traffic, citation, and authority benefits of AI search visibility. This captures the upside while minimizing the downside.

How to Monitor AI Crawler Activity

Once your robots.txt is configured, you need to verify that it is working and track the results. Here are three methods for monitoring AI crawler activity on your website.

Server Access Logs

Your server access logs record every request made to your website, including the user-agent string. Search your logs for the AI crawler user-agents listed in the directory table above. Most hosting panels (cPanel, Plesk, Kinsta) provide access to raw logs or parsed log viewers.

Key metrics to track from your server logs:

• Request volume per bot: How many requests each AI crawler makes per day/week
• Pages accessed: Which pages AI crawlers visit most frequently
• Response codes: Are your robots.txt rules working? Blocked bots should stop visiting blocked paths (though they may still request robots.txt itself)
• Bandwidth consumed: How much server bandwidth AI crawlers are using

GA4 Referral Traffic

In Google Analytics 4, navigate to Reports > Acquisition > Traffic Acquisition and filter by source to identify AI-driven referral traffic. Look for these domains:

• chatgpt.com — Traffic from ChatGPT's cited source links
• perplexity.ai — Traffic from Perplexity's numbered citations
• claude.ai — Traffic from Claude's web search citations
• bing.com/chat — Traffic from Bing Copilot

Create a custom "AI Search" channel group in GA4 that aggregates all AI referral sources. This gives you a single KPI to track over time: "How much traffic am I receiving from AI platforms?" If this number drops to zero after implementing robots.txt changes, you may have accidentally blocked AI search crawlers alongside training crawlers.

robots.txt Validation

Regularly validate your robots.txt file to ensure it is syntactically correct and producing the intended results:

• Google Search Console: Use the robots.txt tester to verify which URLs are blocked for Googlebot and Google-Extended
• seoscore.tools: Our scanner checks your robots.txt configuration as part of its 136+ SEO, AEO, and GEO checks, including specific analysis of AI crawler rules
• Manual testing: Regularly visit your robots.txt file directly (yourdomain.com/robots.txt) to verify the file is accessible and correctly formatted

Frequently Asked Questions

Key Takeaways

1. Your robots.txt is your first line of defense against AI content harvesting. Without specific AI crawler rules, your content is available to every AI training and search bot on the internet. Over 73% of websites have no AI-specific rules — do not be one of them.
2. Distinguish between AI training bots and AI search bots. GPTBot, CCBot, and Bytespider take content for training with no traffic in return. ChatGPT-User, PerplexityBot, and ClaudeBot provide citations and referral traffic. Block the first group, consider allowing the second.
3. The Selective Allow strategy is optimal for most businesses. Block training crawlers (GPTBot, CCBot, Bytespider, anthropic-ai, Google-Extended, FacebookBot, cohere-ai) while allowing search crawlers (ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot). This protects your IP while maintaining AI search visibility.
4. Never block Googlebot. Blocking Googlebot removes you from Google Search entirely. Use Google-Extended to control Gemini AI training without affecting your search rankings or AI Overview visibility.
5. robots.txt is voluntary, not enforceable. Legitimate companies respect it, but rogue scrapers may not. Combine robots.txt with meta robots tags, X-Robots-Tag headers, Terms of Service, and server-side rate limiting for comprehensive protection.
6. Monitor your results. Track AI referral traffic in GA4 (chatgpt.com, perplexity.ai, claude.ai sources), review server logs for AI bot activity, and validate your robots.txt configuration regularly. Use seoscore.tools to audit your AI crawlability across 136+ checks.
7. Update your strategy as the landscape evolves. New AI crawlers emerge regularly. New standards like ai.txt and TDM policies are developing. Review and update your robots.txt configuration at least quarterly to stay current.