AI Bot Directives
What are AI bot directives and how do you control which AI crawlers access your content?
What is ai bot directives?
AI bot directives are rules you set in robots.txt and meta robots tags to control how AI company crawlers (GPTBot, Google-Extended, ClaudeBot, Bytespider, and others) access and use your content. These directives let you decide whether your pages can be crawled for AI training data, used in AI search results, or blocked entirely from specific AI systems.
AI bot directives are rules set in robots.txt and meta robots tags to control how AI company crawlers such as GPTBot, Google-Extended, ClaudeBot, and PerplexityBot access and use your content. They let you decide whether pages can be crawled for AI training data, used in AI search results, or blocked entirely from specific AI systems.
Why does ai bot directives matter?
As AI search engines become major traffic sources, controlling AI bot access is a critical strategic decision. Blocking all AI crawlers means your content will not appear in AI-generated answers, potentially costing significant traffic. Allowing all AI crawlers gives away your content for training without guaranteed attribution. The right approach is selective — allow AI search crawlers that drive traffic and cite sources while potentially restricting pure training crawlers that do not send users back to your site.
As of 2024, over 35% of the top 1,000 websites have added specific AI bot directives to their robots.txt files.
AI-driven search engines now account for an estimated 10-15% of referral traffic for content-heavy websites.
How to fix it
- Identify which AI crawlers are accessing your site by reviewing your server logs for user agents like GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot, and PerplexityBot.
- Add explicit Allow or Disallow rules for each AI bot in your robots.txt file based on your content strategy and whether you want to appear in their AI answers.
- Consider allowing AI search crawlers (GPTBot for ChatGPT search, PerplexityBot for Perplexity) that cite and link to sources while evaluating pure training crawlers on a case-by-case basis.
- Use meta robots or X-Robots-Tag headers for page-level AI bot control when you want different rules for different sections of your site.
- Monitor the evolving AI crawler landscape — new bots appear frequently, and their behavior (search vs. training) may change.
Bad seed, good seed.
# robots.txt — blocks ALL bots including AI search crawlers
User-agent: *
Disallow: /
# You are invisible to both traditional and AI search engines# robots.txt — strategic AI bot management
User-agent: GPTBot
Allow: /blog/
Allow: /learn/
Disallow: /app/
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Disallow: /Frequently asked questions
Should I block AI crawlers from my site?
It depends on your strategy. Blocking AI search crawlers (GPTBot, PerplexityBot) means your content will not appear in AI-generated answers, costing traffic. Consider selective blocking — allow AI search crawlers that cite sources and block pure training crawlers.
What is the difference between GPTBot and Google-Extended?
GPTBot is OpenAI's crawler used for both ChatGPT search and model training. Google-Extended controls Google's use of your content for AI features like Gemini but does not affect regular Google Search indexing.
Do AI crawlers respect robots.txt?
Major AI crawlers from OpenAI, Google, Anthropic, and Perplexity have committed to respecting robots.txt. However, compliance is voluntary and not all crawlers may honor your directives.
Related guides
Robots Meta
The robots meta tag is an HTML <meta name="robots"> element that instructs search engine crawlers whether to index a page and whether to fol...
Read more → AI SEOStructured Data
Structured data is machine-readable markup (typically JSON-LD using the Schema.org vocabulary) embedded in your page's HTML that explicitly...
Read more → AI SEOContent Extractability
Content extractability measures how easily AI systems and web crawlers can parse, understand, and pull meaningful information from your page...
Read more →Check your AI search visibility
Find and fix these issues automatically with eiSEO.
50 pages free · No credit card · Results in under 2 minutes