Back to Glossary
Definition
robots.txt for AI Crawlers
robots.txt is a standard file that tells crawlers what they're allowed to access; OpenAI provides user agents like GPTBot and OAI-SearchBot that can be managed via robots.txt.
Why robots.txt for AI Crawlers Matters
Block the wrong paths and your best "source pages" won't be available to crawlers, reducing your AI visibility.
Key Factors
robots.txt configuration directly controls which AI crawlers can access your content and which pages they can index.
Action Steps
Confirm key pages aren't disallowed to relevant bots. Keep XML sitemaps and internal links healthy. Avoid blocking docs, pricing, comparisons, and definitions pages by accident. Audit AI crawler access with BeRecommended.com.