AI Visibility
AI visibility is about controlling how AI search engines like ChatGPT, Perplexity, and Google AI Overviews discover and use your content. Proper configuration of robots.txt and llms.txt ensures your site is accessible to the AI crawlers that drive modern search traffic.
Robots.txt for AI Search Engines
Robots.txt is a text file at the root of your website that tells web crawlers which pages they can and cannot access. With the rise of AI search engines, robots.txt has become the primary way to control whether AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), and PerplexityBot (Perplexity) can crawl your content. Each AI company has its own crawler user-agent, and you need specific directives for each one to control access.
llms.txt
llms.txt is a proposed standard file (placed at /llms.txt on your domain) that provides a structured, plain-text summary of your website specifically for large language models. While robots.txt controls whether AI crawlers can access your site, llms.txt helps them understand what your site is about, what content is most important, and how it's organized. Think of it as a "readme" for AI — a concise document that gives AI models the context they need to accurately represent your content in search results.
Audit your AI crawler access now
No credit card required. Start scanning in under two minutes.