TL;DR

A peer-reviewed study from Wharton and Rutgers found that news publishers who blocked AI crawlers via robots.txt lost approximately 7% of their weekly traffic within six weeks — with no measurable content-protection benefit. The data confirms what many SEO professionals suspected: blocking AI bots does not keep your content out of AI answers; it just removes your brand from the citation.

The Study: What the Numbers Say

Researchers Hangcheng Zhao (Wharton) and Ron Berman (Wharton/Rutgers) analyzed publisher behavior and traffic data across SimilarWeb, Semrush, Comscore, HTTP Archive, Wayback Machine, and Revelio Labs covering November 2022 through May 2024 — the period when most publishers began adding GPTBot, ClaudeBot, PerplexityBot, and Google-Extended to their robots.txt files.

The headline finding from their paper "Strategic Response of News Publishers to Generative AI" (SSRN, revised April 2026): publishers who blocked LLM crawlers saw a ~7% decline in weekly traffic in human browsing panels within six weeks of implementing the block.

An earlier draft of the same study reported an even steeper figure — roughly 23% monthly visit loss. The revised, more conservative 7% weekly number accounts for confounding variables and seasonal effects, making it a more defensible benchmark. Even at 7%, the business impact is substantial: a publisher with 1 million weekly visits loses 70,000 visits — and the ad revenue attached to them — every single week.

Why "Content Protection" Is an Illusion

The study's second finding may be even more significant than the traffic loss: blocking AI crawlers provided no measurable content-protection benefit.

This makes intuitive sense when you consider how large language models actually acquire information. Even if you block GPTBot from crawling your site directly, your content still reaches AI models through multiple other channels: Common Crawl archives, Reddit discussions that quote or summarize your articles, syndication partners who republish your work, and cached versions from before the block took effect.

The result is that AI systems continue to answer questions your content originally addressed — they just do so without citing you as the source. You lose the attribution, the referral traffic, and the brand visibility, while gaining nothing in return.

What to Do Instead: A 3-Step Framework

If blocking AI crawlers is counterproductive, what should brands and publishers do? The answer lies in shifting from a defensive posture to an active visibility strategy.

Step 1: Monitor your AI citation share. Before you can optimize anything, you need to know where you stand. Track how often your brand appears in AI-generated answers across ChatGPT, Perplexity, Gemini, and Google AI Overviews. Tools like Be Recommended can measure your citation footprint across multiple AI platforms and show you which queries mention your brand — and which ones should but don't.

Step 2: Optimize your content for AI citation. AI models favor content that is clearly structured, factually dense, and easy to attribute. Use clear entity markup (JSON-LD for Organization, Product, FAQ), write in a question-and-answer format where appropriate, and ensure your most important claims are supported by data and sources. Think of it as "snippet optimization" for AI, not just for Google's featured snippets.

Step 3: Build presence on AI-cited platforms. Recent research shows that AI search engines pull citations disproportionately from platforms like YouTube, Reddit, LinkedIn, and Wikipedia. Publishing thought leadership on these platforms — in addition to your own site — creates multiple citation vectors that AI models can draw from, all pointing back to your brand.

The Bigger Picture

This study aligns with other recent findings that paint a consistent picture: AI-driven traffic is not something to fear or block — it is something to actively cultivate.

Research from Adobe showed that AI-referred visitors convert at 42% higher rates than traditional search visitors. A Seer Interactive analysis of AI Overviews found that brands cited in AI answers see roughly 35% higher click-through rates on adjacent organic results.

The pattern is clear: brands that show up in AI answers win twice — they get direct referral traffic from the AI platform and a halo effect on their traditional search performance.

Key Takeaways

Blocking AI crawlers via robots.txt costs publishers ~7% weekly traffic with zero content-protection upside
AI models still access your content through alternative channels (Common Crawl, Reddit, syndication), so blocking only removes your citation credit
The alternative: monitor your AI citation share, optimize content for AI readability, and build presence on platforms AI models cite most
AI-referred traffic converts better and boosts adjacent organic CTR — it is a channel worth investing in, not blocking

FAQ

Should I remove AI crawler blocks from my robots.txt right now? If you have blocks in place, the study suggests removing them is likely net-positive. However, audit your analytics first to understand your current baseline, so you can measure the impact of the change.

Does this apply to e-commerce sites or just publishers? The study focused on news publishers, but the underlying mechanics apply broadly. Any site that blocks AI crawlers risks losing citation credit and referral traffic from AI platforms.

How do I know if I'm being cited in AI answers? Manually querying ChatGPT and Perplexity about your brand or topics is a start. For systematic monitoring, Be Recommended tracks your brand mentions across major AI platforms automatically.

Sources

PPC Land: Blocking AI Crawlers Cost News Publishers 7% of Traffic, Study Finds (April 26, 2026)
Noah News: Blocking AI Crawlers Costs News Publishers 7% of Weekly Traffic, Study Finds (April 26, 2026)
Zhao, H. & Berman, R. "Strategic Response of News Publishers to Generative AI" — SSRN (v3, revised April 21, 2026)
Adobe Digital Economy Index: AI-Driven Traffic in E-commerce (2024)
Seer Interactive: AI Overviews Research Study

Blocking AI Crawlers Costs Publishers 7% of Traffic — Here's What to Do Instead