Robots.txt for AI Search Engines
How to configure robots.txt for AI search engines like ChatGPT, Perplexity, and Claude
What is robots.txt for ai search engines?
Robots.txt is a text file at the root of your website that tells web crawlers which pages they can and cannot access. With the rise of AI search engines, robots.txt has become the primary way to control whether AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), and PerplexityBot (Perplexity) can crawl your content. Each AI company has its own crawler user-agent, and you need specific directives for each one to control access.
Robots.txt for AI search engines is the practice of adding explicit User-agent directives for AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and others — to your robots.txt file, controlling whether each AI-powered search platform can crawl and use your content in AI-generated answers.
Why does robots.txt for ai search engines matter?
AI-powered search engines are rapidly becoming a primary way people find information. ChatGPT, Perplexity, and Google AI Overviews collectively handle billions of queries. If your robots.txt blocks these crawlers — or doesn't mention them at all — your content won't appear in AI-generated answers. Many websites inadvertently block AI crawlers with overly restrictive wildcard rules, losing visibility in the fastest-growing search channel without realizing it.
Over 45% of the top 1,000 websites now block at least one AI crawler via robots.txt, up from less than 5% in 2023.
ChatGPT alone processes over 1 billion queries per week, making AI search a significant traffic source for websites that allow crawler access.
How to fix it
- Check your current robots.txt file by visiting yourdomain.com/robots.txt and look for any User-agent directives that mention AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, CCBot, anthropic-ai, FacebookBot).
- If you have a wildcard "Disallow: /" rule for User-agent: *, add explicit "Allow: /" directives for each AI crawler you want to permit.
- Add separate User-agent blocks for each AI crawler with the appropriate Allow or Disallow directives based on your content strategy.
- Test your changes using a robots.txt testing tool or the free eiSEO AI Crawler Audit at /tools/ai-crawler-audit to verify each crawler's access status.
- Review your robots.txt quarterly as new AI crawlers emerge — the AI search landscape is changing rapidly.
Bad seed, good seed.
# Bad: No AI-specific directives
# AI crawlers fall through to wildcard
User-agent: *
Disallow: /private/
Disallow: /admin/
# Result: AI crawlers have access, but you
# have no granular control over which ones# Good: Explicit AI crawler directives
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: CCBot
Allow: /
User-agent: *
Disallow: /private/
Disallow: /admin/Frequently asked questions
Which AI crawlers should I allow in robots.txt?
At minimum, allow GPTBot (OpenAI/ChatGPT), ChatGPT-User (ChatGPT live browsing), Google-Extended (Google AI Overviews), ClaudeBot (Anthropic/Claude), and PerplexityBot (Perplexity search). These represent the largest AI search platforms. Also consider CCBot (Common Crawl, used to train many models) and anthropic-ai (secondary Anthropic crawler).
Does blocking AI crawlers in robots.txt also block traditional search?
No. AI crawler user-agents are separate from traditional search bot user-agents (Googlebot, Bingbot). Blocking GPTBot does not affect your Google organic rankings. However, blocking Google-Extended does prevent Google from using your content in AI Overviews while keeping your regular search listings unaffected.
How often should I update my robots.txt for AI crawlers?
Review quarterly. The AI search landscape is evolving rapidly, with new crawlers appearing regularly. Major events like new AI search products (e.g., when Perplexity launched, or when OpenAI released SearchGPT) introduce new user-agents that need explicit rules.
Related guides
llms.txt
llms.txt is a proposed standard file (placed at /llms.txt on your domain) that provides a structured, plain-text summary of your website spe...
Read more → AI SEOStructured Data
Structured data is machine-readable markup (typically JSON-LD using the Schema.org vocabulary) embedded in your page's HTML that explicitly...
Read more → AI SEOContent Extractability
Content extractability measures how easily AI systems and web crawlers can parse, understand, and pull meaningful information from your page...
Read more →Check your AI crawler access with our free audit tool
Find and fix these issues automatically with eiSEO.
50 pages free · No credit card · Results in under 2 minutes