AI Crawler Audit
Enter any domain to see which AI search engines can access your content. Check your robots.txt and llms.txt configuration instantly.
Run the audit
Four crawlers, two files, ten seconds.
The AI Crawler Audit scans your domain's robots.txt file and checks for the presence of an llms.txt file. It reports which major AI crawlers are allowed or blocked, so you can make informed decisions about your AI search visibility.
GPTBot
OpenAI's crawler used to gather training data and power ChatGPT search results.
ClaudeBot
Anthropic's web crawler for Claude's knowledge and search capabilities.
PerplexityBot
Perplexity AI's crawler that indexes content for its answer engine.
Google-Extended
Google's AI training crawler, separate from the standard Googlebot used for search indexing.
Why it matters
If your robots.txt blocks these crawlers, your content won't appear in AI-generated answers, even if it ranks well in traditional search. Many sites unknowingly block AI bots through overly broad robots.txt rules. This tool helps you identify and fix those gaps in seconds.
Go deeper on AI visibility.
Want continuous AI visibility monitoring?
eiSEO monitors your AI search visibility across all major platforms and alerts you when crawlers are blocked.
50 pages free · No credit card · Results in under 2 minutes