EN
Free audit

Which AI bots should you allow in robots.txt?

Updated 4 min read

Short answer

Separate AI search bots from training bots. Allow the bots that fetch pages for answers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User), otherwise assistants can't cite you. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot and others) are a business decision. Then check that your firewall doesn't block what robots.txt allows.

Why does robots.txt matter for AI visibility?

When an AI assistant searches the web to answer a question, it can only quote pages its crawler is allowed to fetch. A rule copied from a "block all AI" template can remove your site from ChatGPT, Claude or Perplexity answers while your competitors stay in them. The opposite mistake also happens: a site wants to stop training on its content and accidentally blocks the search bots too.

What are the main AI user agents?

CompanyUser agentPurpose
OpenAIOAI-SearchBotIndex for ChatGPT search
OpenAIChatGPT-UserFetches pages on a user's request
OpenAIGPTBotTraining data
AnthropicClaude-SearchBotSearch index for Claude
AnthropicClaude-UserFetches pages on a user's request
AnthropicClaudeBotTraining data
PerplexityPerplexityBotPerplexity's search index
PerplexityPerplexity-UserFetches pages on a user's request
GoogleGoogle-ExtendedA robots.txt token for Gemini training and some Gemini grounding; doesn't affect Google Search or AI Overviews
AppleApplebot-ExtendedA token to opt out of Apple's AI training; Applebot itself is Apple's search crawler
Common CrawlCCBotOpen web dataset used to train many models
MetaMeta-ExternalAgentTraining data
AmazonAmazonbotAlexa and other Amazon services
ByteDanceBytespiderTraining data

Google's AI Overviews and AI Mode use the normal Googlebot. Blocking Google-Extended doesn't remove you from them; blocking Googlebot removes you from Google entirely.

Which bots should you allow?

Allow search and user bots if you want AI assistants to cite and recommend you. They fetch pages when somebody is asking a question right now, which is exactly the moment you want to be found.

Decide on training bots based on your business. A store or a service company usually benefits from being well known to models, so allowing training is reasonable. A publisher whose content is the product may prefer to block training. There is no single right answer, but it should be a decision, not an accident.

A ready robots.txt: AI search open, training closed

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /cart
Disallow: /admin/

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /cart
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml

Two details people miss:

  1. **A specific group replaces the group.* If you create a group for OAI-SearchBot with only Allow: /, that bot ignores your Disallow: /cart rules from *. Copy the service restrictions into the AI search group, as above.
  2. Several user agents can share one group by listing them one under another before the rules.

To open everything, put all AI bots in one group with your normal restrictions and set ai-train=yes.

What is Content-Signal?

Content-Signal is a line introduced by Cloudflare in its Content Signals Policy in 2025. It expresses how content may be used after it is fetched: search for search indexes and links, ai-input for using content in AI answers, ai-train for model training. It is a statement of preference, not a technical block, and AI companies are not obliged to honour it. Adding it costs nothing and makes your intent explicit.

Why do bots still get blocked when robots.txt allows them?

Because robots.txt is only a request. The actual gatekeeper is your server, CDN or firewall. Common causes:

  • bot protection or "block AI scrapers" switches in CDN settings;
  • WAF rules that challenge unknown user agents with a captcha;
  • rate limits that return 429 or 403 to crawlers;
  • hosting providers that block whole data-centre IP ranges.

Check your firewall logs for these bots, or compare how your homepage responds to a normal browser and to an AI bot user agent. Our free audit does such a comparison for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Keep in mind that some protection services verify bots by IP address, so a test with a borrowed user agent can be blocked even when the real bot is allowed. The logs are the final word.

Is robots.txt enough to stop training?

Reputable AI companies say their crawlers respect robots.txt, but it isn't enforcement. If preventing training is critical, combine robots.txt with firewall rules for the training bots and keep your terms of use explicit.

What else should you check?

  • noindex or noai in meta robots or the X-Robots-Tag header on key pages.
  • Text that only appears after JavaScript runs (JavaScript and AI crawlers).
  • A sitemap linked from robots.txt.
  • An llms.txt that opens as a real file, not a 404 page (llms.txt explained).

The AI-Ready Pack generates a robots.txt like the one above from your existing file, keeping your current rules and replacing only the AI section.