Google Contradicts Itself on llms.txt: Lighthouse Audits What Search Central Says to Skip

TL;DR

Google's two teams now give opposite advice on llms.txt: Search Central says skip it, Chrome Lighthouse 13.3 audits it by default. The right answer depends on your audience — AI Search users or agentic tool users — and this decision framework helps you choose.

Key Takeaways

• Google Search Central's AI optimization guide explicitly lists llms.txt among tactics brands can skip for AI Overviews and AI Mode
• Chrome Lighthouse 13.3 (default since May 2026) added an Agentic Browsing category that checks for llms.txt as part of website readiness
• The contradiction reflects two different use cases: Google Search AI features versus browser-based AI agents
• Brands need a decision framework based on traffic mix, Lighthouse reporting requirements, content change frequency, and risk appetite
• Server log data shows that 99% of LLM bot traffic never requests llms.txt, making the file irrelevant for most sites — unless you serve a developer or IDE audience

What Happened

Two Google teams released conflicting guidance on llms.txt within days of each other — and neither seems aware of the contradiction.

On May 15, Google Search Central published its first official AI optimization guide. The document is direct: brands do not need llms.txt files, special AI markup, content chunking, or machine-readable Markdown to appear in AI Overviews or AI Mode. John Mueller had previously compared llms.txt to the keywords meta tag — a well-intentioned idea that search engines ultimately ignore.

Then Lighthouse 13.3 shipped on May 7 with a new Agentic Browsing category enabled by default. Among its four audits: an llms.txt check. If your site lacks the file, Lighthouse notes that agents may need extra time to understand your site structure. The category does not produce a single 0–100 score — the Lighthouse team chose actionable signals over rankings because agentic web standards are still emerging — but it does flag missing files.

To add to the confusion, Google's own developer.chrome.com and web.dev domains still serve llms.txt files. When Lidia Infante spotted a similar file on Search Central's own docs site in December 2025, Mueller responded with a sheepish reaction and the file disappeared within hours. Different internal CMS teams, different standards.

Why It Matters for AI Visibility

The contradiction is not a bug — it reflects two genuinely different use cases that most brands conflate.

Google Search Central's advice covers one pipeline: how content appears in AI Overviews and AI Mode. These features pull from Google's core ranking systems using RAG and query fan-out. They do not consult llms.txt. For this use case, the file adds nothing.

Lighthouse's Agentic Browsing audits target a different pipeline entirely: how browser-based AI agents — think IDE assistants, MCP-connected tools, and autonomous research agents — interact with your website. These agents read the accessibility tree, look for WebMCP integration, and yes, check whether llms.txt provides a machine-readable summary of your site.

As we covered when Google first published its AI guide, the advice applies strictly to Google's own AI features. ChatGPT, Perplexity, Claude, and Gemini API each have different retrieval mechanisms. The real question is not whether llms.txt matters for Google Search — it does not — but whether it matters for the other seven retrieval layers that now sit between your content and AI-generated answers.

For most brands today, the honest answer is: probably not yet. As our earlier analysis of llms.txt adoption explored, the spec has vocal advocates but thin real-world traction. Analysis of over 515 million LLM bot events shows that virtually no AI crawlers request llms.txt. The file exists as a spec with growing tooling support but minimal real-world consumption. That could change as agentic browsing matures, which is exactly why Lighthouse started measuring it now.

What Brands Should Do Now

Rather than following one Google team over another, use this four-question framework:

1. What is your traffic mix? If your audience includes developers, AI tool builders, or IDE-adjacent users who interact with your content through agents, implement llms.txt now. Browser agents and MCP-connected tools are the primary consumers. If your traffic is pure consumer searching via Google or ChatGPT, skip it — no engine currently uses it for answer generation.

2. Do you report Lighthouse scores? If your team or clients track Lighthouse as a deliverable or KPI, add a basic llms.txt to clear the audit. The file takes five minutes to create and eliminates a warning from a default audit category.

3. How often does your content change? A stale llms.txt that lists deprecated pages or wrong navigation is worse than no file at all. If your site structure changes weekly, the maintenance cost may outweigh the benefit until automated generation tools mature.

4. What is your risk profile? Early adopters who want to be ready for the agentic web should implement llms.txt now — the llms.txt specification is straightforward. Conservative teams should wait until crawler logs show meaningful request volume for the file on their own domain.

How to Measure Impact

Start by checking your current Lighthouse Agentic Browsing results — run npm install -g lighthouse@latest and test your site, or use an online tool like DebugBear's website checker. The category shows actionable signals rather than a pass/fail score.

Next, monitor your server logs for requests to /llms.txt. If you see near-zero volume (which is the norm for most sites in mid-2026), that confirms the file is not yet a priority for your audience. If you see requests from specific bot user agents, investigate which platforms are consuming it.

For broader AI visibility, track how your brand appears across multiple AI platforms — not just Google. The gap between what Google recommends for its own features and what actually drives citations in ChatGPT, Perplexity, or Claude is where the real optimization opportunity lives. Tools like Be Recommended measure cross-platform AI visibility independent of any single engine's guidance.

The llms.txt debate will keep evolving throughout 2026. What matters now is making an informed choice based on your specific audience, not following contradictory guidance from teams that appear to be talking past each other.

FAQ

Do I need llms.txt for Google AI Overviews? No. Google Search Central's May 2026 guide explicitly states that llms.txt is not needed for AI Overviews or AI Mode. Google's AI features rely on core Search ranking systems, not on llms.txt.

Will missing llms.txt hurt my Lighthouse score? Not exactly. The Agentic Browsing category does not ship a weighted 0–100 score. If the file is absent, the audit is marked as "Not Applicable" rather than a failure. However, if you serve the file with errors (server 500, missing H1), it will be flagged.

Should I add llms.txt just to be safe? Only if it fits your traffic and maintenance reality. A well-maintained llms.txt costs almost nothing. A stale one with outdated links can mislead agents. Use the four-question framework above to decide.

Which AI platforms actually read llms.txt? As of mid-2026, virtually none at meaningful scale. Server log analyses show near-zero request rates from major AI crawlers. The specification has growing tooling support (Lighthouse, Wix, various CMS plugins) but limited real-world consumption by AI retrieval systems.

Sources

• Search Engine Journal — Google's llms.txt Guidance Depends On Which Product You Ask
• Chrome for Developers — llms.txt Lighthouse audit
• DebugBear — Google Lighthouse Has A New Agentic Browsing Category
• Search Engine Land — Google adds llms.txt check to Chrome Lighthouse
• Google Search Central — AI optimization guide
• Chrome for Developers — Lighthouse Agentic Browsing scoring