Robots Meta
What is the robots meta tag and how does it control search engine indexing?
What is robots meta?
The robots meta tag is an HTML <meta name="robots"> element that instructs search engine crawlers whether to index a page and whether to follow its links. Common directives include index/noindex (whether to include the page in search results) and follow/nofollow (whether to pass link equity through the page's outbound links). An X-Robots-Tag HTTP header can also deliver these directives.
The robots meta tag is an HTML meta element that instructs search engine crawlers whether to index a page and whether to follow its links, using directives such as index, noindex, follow, and nofollow. An accidental noindex tag is one of the most devastating SEO mistakes because it completely removes a page from search results.
Why does robots meta matter?
An accidental noindex tag is one of the most devastating SEO mistakes possible — it completely removes your page from search results. This commonly happens when a noindex tag is added during development or staging and not removed before launch. Conversely, pages that should not be indexed (admin panels, thank-you pages, internal search results) can waste crawl budget and create thin-content issues if they lack a noindex directive.
Key statistics
An estimated 25% of indexable pages on the web have robots meta tag issues including conflicting directives or unintended noindex tags.
Source: Screaming Frog
How to fix it
-
1
Audit every page for unintended noindex directives. Check both the <meta name="robots"> tag and X-Robots-Tag HTTP headers.
-
2
Add noindex to pages that should not appear in search results: admin pages, staging environments, internal search results, duplicate paginated pages, and thin utility pages.
-
3
Ensure your production environment does not carry over noindex tags from staging. Add a deployment check to verify critical pages are indexable.
-
4
Use Google Search Console's URL Inspection tool to verify how Google sees your robots directives for specific pages.
Code example
<!-- Left over from staging — this removes the page from Google entirely! -->
<meta name="robots" content="noindex, nofollow"><!-- For indexable pages: explicitly allow indexing -->
<meta name="robots" content="index, follow">
<!-- For pages that should NOT be indexed (e.g., /thank-you): -->
<meta name="robots" content="noindex, nofollow">Frequently asked questions
Related topics
Canonical Tags
A canonical tag (rel="canonical") is an HTML element placed in the <head> of a page that tells search engines which URL is the preferred or "canonical" version of that page. When the same content is accessible at multiple URLs (due to query parameters, www/non-www variations, HTTP/HTTPS, or syndication), the canonical tag consolidates all ranking signals to a single URL.
AI Bot Directives
AI bot directives are rules you set in robots.txt and meta robots tags to control how AI company crawlers (GPTBot, Google-Extended, ClaudeBot, Bytespider, and others) access and use your content. These directives let you decide whether your pages can be crawled for AI training data, used in AI search results, or blocked entirely from specific AI systems.
Title Tags
The title tag is the HTML <title> element that defines the page's title in the browser tab and, most importantly, as the clickable headline in search engine results pages (SERPs). It is widely considered the single most important on-page SEO element because it directly tells search engines and users what the page is about.
Analyze your site's SEO health
eiSEO automatically detects and helps you fix issues like this across your entire site.