Back to Glossary
Definition
X-Robots-Tag
X-Robots-Tag is an HTTP header that can instruct crawlers how to handle indexing (including noindex).
Why X-Robots-Tag Matters
If important pages are unintentionally blocked from indexing/discovery, they're far less likely to be retrieved and cited in AI search experiences that rely on web access.
Key Factors
- Audit header rules sitewide (especially staging → production transitions)
- Keep "source pages" indexable (glossary, comparisons, definitions)
- Use noindex only for pages you truly don't want visible
Action Steps
Run a technical audit to ensure your key pages don't have unintentional noindex headers from staging configurations.