Free tool — no signup required
AI Crawler Audit
Enter any domain to see which AI search engines can access your content. Check your robots.txt and llms.txt configuration instantly.
What This Tool Checks
The AI Crawler Audit scans your domain's robots.txt file and checks for the presence of an llms.txt file. It reports which major AI crawlers are allowed or blocked, so you can make informed decisions about your AI search visibility.
AI Crawlers We Check
- GPTBot — OpenAI's crawler used to gather training data and power ChatGPT search results.
- ClaudeBot — Anthropic's web crawler for Claude's knowledge and search capabilities.
- PerplexityBot — Perplexity AI's crawler that indexes content for its answer engine.
- Google-Extended — Google's AI training crawler, separate from the standard Googlebot used for search indexing.
Why It Matters
If your robots.txt blocks these crawlers, your content won't appear in AI-generated answers — even if it ranks well in traditional search. Many sites unknowingly block AI bots through overly broad robots.txt rules. This tool helps you identify and fix those gaps in seconds.
Learn more about AI visibility
Want continuous AI visibility monitoring?
eiSEO monitors your AI search visibility across all major platforms and alerts you when crawlers are blocked.