Cloudflare Radar's 28-day window ending May 18, 2026 confirmed a reversal: GPTBot now holds 11.97% of AI bot HTTP requests versus ClaudeBot's 10.67%. Combined with OAI-SearchBot, OpenAI crawlers account for 13.64% of all AI crawl traffic, putting them clearly ahead of Anthropic for the first time. If you assumed your SEO rankings meant equal visibility across all AI engines, this data says otherwise. Each crawler sees a different slice of the web, and your brand may be present in ChatGPT answers while completely absent from Claude.

TL;DR

GPTBot overtook ClaudeBot in May 2026 crawl share (11.97% vs 10.67%), reversing an April lead by Anthropic. The crawl-to-referral asymmetry is extreme: ClaudeBot crawls 13,528 pages for every single referral it sends back, while GPTBot sits at 1,276:1. Training accounts for 52.54% of all AI crawl activity; actual search is just 8.41%. For brands, this means AI visibility is not one metric. It is a per-engine measurement that requires per-engine strategy.

What Happened

Cloudflare's latest Radar data captured a shift in the AI crawler hierarchy. After ClaudeBot briefly led in April 2026 (11.69% vs GPTBot's 9.84%), OpenAI's training crawler surged back to 11.97% in the May window. Anthropic's training burst in April, likely tied to the Claude Opus 4.7 release on April 16, faded as OpenAI ramped its own collection.

The more telling number is the crawl-to-referral ratio. According to Cloudflare's analysis, ClaudeBot crawled 13,528 pages for every one click it sent back to publishers. GPTBot's ratio: 1,276:1. DuckDuckGo, by comparison, operates at near parity (1.5:1). A new entrant also appeared: Claude-SearchBot hit 2.00% share in May, marking the first time Anthropic's dedicated search crawler was visible in publisher telemetry.

On purpose breakdown, training dominates at 52.54% of all AI crawl traffic (up from 50.69% in April). Mixed-purpose crawling accounts for 36.06%. Search, the only category that directly generates referral traffic, represents just 8.41%.

Why It Matters for AI Visibility

The headline number (GPTBot vs ClaudeBot share) is interesting for the SEO community. The real implication for brand managers is different: each AI engine builds its knowledge from a different crawler with different frequency, coverage, and site selection logic.

A brand that is well-indexed by GPTBot will appear in ChatGPT answers. That same brand may be partially or completely invisible in Claude's responses if ClaudeBot has not crawled the relevant pages recently or thoroughly. Googlebot, still the largest single crawler at roughly 39% of combined AI and search traffic, feeds Google's AI Overviews through yet another pipeline. Three engines, three crawlers, three different snapshots of your brand.

This matters because blocking decisions compound the problem. According to a TechnologyChecker.io analysis of robots.txt files across Cloudflare's network, 69% of websites block ClaudeBot and 62% block GPTBot. The retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot) get blocked at lower but still significant rates: 49% for OAI-SearchBot, 40% for ChatGPT-User. Every block is a deliberate removal of your brand from that engine's citation surface.

The practical takeaway: "good SEO" no longer equals "visible in AI." Your robots.txt policy, your crawl frequency per bot, and your content's accessibility to each crawler collectively determine whether you get cited when someone asks ChatGPT, Claude, or Perplexity about your category.

What Brands Should Do Now

Audit your robots.txt per AI bot. Check whether your current rules block training crawlers (GPTBot, ClaudeBot) and retrieval crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) separately. Blocking a training bot removes your content from the next model version. Blocking a retrieval bot removes you from live AI search results. These are different decisions with different business implications.

Run a server log analysis by user-agent. Filter your access logs for GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, and Google-Extended. Map which pages each bot actually visits, how frequently, and whether your key product and content pages are getting crawled across all engines. If ClaudeBot visits your homepage but skips your product pages, Claude will not cite you for product queries.

Implement per-bot content prioritization. If you run a segmented llms.txt file (the emerging standard for telling AI crawlers which content matters most), ensure it covers the pages you want each engine to see. Your crawl budget per bot is finite; point it at high-value pages.

Test your actual AI visibility per engine. Send 5 to 10 queries about your brand and category to ChatGPT, Claude, Perplexity, and Gemini. Note which platforms cite you, which cite competitors, and which produce answers without citing anyone. This 30-minute exercise reveals your real per-engine visibility baseline more accurately than any aggregate metric.

Monitor the Cloudflare AI Crawlers dashboard. If you use Cloudflare, enable the AI Crawlers analytics to see per-bot request volume in real time. This is the fastest way to confirm whether a robots.txt change or content update is actually being picked up by each crawler.

How to Measure Impact

Start with three per-engine metrics tracked weekly:

Crawl frequency per bot. Server logs tell you how often each AI crawler visits your key pages. A page that GPTBot hits daily but ClaudeBot visits monthly will have fresh representation in ChatGPT but stale (or absent) representation in Claude.

Citation rate per engine. Use a tool like BeRecommended to measure how often your brand appears in AI answers across ChatGPT, Claude, Perplexity, and Gemini for your top queries. Compare engines side by side. A 40% citation rate in ChatGPT with 5% in Claude is a clear signal that your content is not reaching Anthropic's pipeline.

Referral traffic by AI source. In GA4, segment referral traffic by AI platform. If you see ChatGPT referrals growing while Claude referrals stay at zero, the crawl data explains why: ClaudeBot's 13,528:1 ratio means it sends almost no traffic back even when it does crawl your content.

The brands that treat AI visibility as a single aggregate number will miss the gaps. The ones that measure per engine, diagnose per crawler, and optimize per bot will build compounding visibility across all AI platforms, not just the one that happens to crawl most aggressively this month.

FAQ

Does blocking GPTBot also block ChatGPT from citing me?

Not directly. GPTBot handles training data collection. OAI-SearchBot and ChatGPT-User handle real-time retrieval for search and conversation. You can block GPTBot (training) while allowing OAI-SearchBot (search citations). However, blocking GPTBot means your content will not be included in future model training, which can reduce your long-term organic visibility in ChatGPT answers.

Why did ClaudeBot lead in April but fall behind in May?

Anthopic ran a heavy training data collection burst in April 2026, likely preparing data for the Claude Opus 4.7 release (shipped April 16). Once the training window closed, ClaudeBot's crawl volume normalized. OpenAI's GPTBot, meanwhile, increased its own collection through May.

Is the crawl-to-referral ratio improving for any AI platform?

Yes. Anthropic's ratio dropped from 286,930:1 in January 2025 to 38,065:1 by July 2025, a significant improvement after launching web search in Claude. Claude-SearchBot's appearance in May 2026 suggests further referral improvements are coming. OpenAI's ratio has been relatively stable at around 1,000 to 1,300:1.

How often should I check my per-engine AI visibility?

Weekly for active campaigns or content launches. Monthly for steady-state monitoring. Crawl patterns can shift within weeks (as the April-to-May GPTBot/ClaudeBot reversal showed), so quarterly checks will miss important changes.

Sources

Cloudflare: The crawl-to-click gap: AI bots, training, and referrals (Cloudflare blog, primary source for crawl-to-refer ratios)
Cloudflare: A deeper look at AI crawlers by purpose and industry (purpose breakdown: training vs search)
Search Engine Journal: Cloudflare Report on AI Crawler Traffic (third-party analysis)
Cloudflare Radar AI Insights (live dashboard for AI bot traffic data)

GPTBot Just Passed ClaudeBot in Crawl Share. Your Brand Visibility per AI Engine Depends on Which Bot Sees You First.