llms.txt: Server Logs Say AI Crawlers Aren’t Reading It

What this blog covers

llms.txt was pitched as a manifest that tells AI crawlers what a site is and where its best content lives. A large-scale server-log study of over 137,000 domains found that 97% of published llms.txt files received zero AI-bot requests, while a smaller vendor-side study has claimed a measurable lift from adopting one. This piece lays out both findings honestly and explains what to actually prioritize instead.

Quick Answer: Server-log evidence from a May 2026 study of 137,210 domains found that 97% of published llms.txt files received zero requests from any AI crawler, and that AI systems never proactively looked for files that did not exist. A separate vendor study has reported a positive lift from publishing one. The honest reading: llms.txt costs little to publish. It has no proven mechanism for improving AI citation today, so it belongs at the bottom of a GEO priority list, well below schema, entity consistency, and answer-first content structure.

The Study That Started the Debate

In May 2026, Ahrefs analyzed server logs across 137,210 domains on its Web Analytics platform. 28% of those domains had a valid llms.txt file published. The question the study set out to answer was simple: are AI crawlers actually requesting it?

The scale of the study is what makes it hard to dismiss. This is not a handful of anecdotal sites comparing notes; it is a broad cross-section of the live web, measured directly at the server level rather than inferred from rankings or citation counts.

What the Logs Actually Showed

97% of the llms.txt files that existed received zero requests during the study window. Only 3%, around 1,100 domains, saw any traffic to the file at all, and human visitors accounted for a meaningful share of even that small number. Among AI bots that did request a file, GPTBot was the most active individual crawler, still a small fraction of overall requests.

Perhaps the more telling finding: AI bots never proactively requested a file that did not exist. There is no evidence of crawlers checking for llms.txt as a matter of routine and adjusting behavior when they find one, which undercuts the core promise the format was built on. Brands weighing where to invest their technical GEO budget are better served starting with the fundamentals covered in how to get cited in AI search, which several L&F engagements have validated more directly.

The Counter-Claim, and Why It Doesn’t Resolve the Contradiction

Not every voice in this debate agrees. Vendors selling llms.txt tooling and generation services have published their own data claiming a visibility lift for sites that adopt the file, and some individual practitioners report anecdotal citation improvements after publishing one.

These claims are not necessarily fabricated, however they are methodologically thin next to a 137,000-domain, server-log-verified study. Correlation with a citation increase is easy to produce when a site is also doing other GEO work at the same time (schema cleanup, answer-first rewrites, entity consistency) and the llms.txt file gets the credit. We are not resolving this contradiction here. It exists, and any team making a budget decision should know both sides exist before choosing where the evidence weighs heavier.

Why This Keeps Getting Oversold

llms.txt has the shape of a solution SEO teams want: a single file, a clear spec, a checkbox that feels like control in a landscape that otherwise offers none. That psychological pull is exactly why it spread faster than the evidence for it did.

It also costs almost nothing to publish, which lowers the bar for recommending it. A consultant or tool vendor loses little by suggesting it and gains a talking point; a brand loses little time implementing it. The asymmetry between low cost and uncertain payoff means it keeps appearing on checklists regardless of what the logs say.

What to Prioritize Instead

The GEO fundamentals with actual evidence behind them sit elsewhere: structured entity data, answer-first content structure, and consistent product or company claims across the platforms an AI model can see. These are the layers schema markup for AI search and entity-based SEO work is built around, and they show up in citation behavior in ways llms.txt has not yet demonstrated. The deeper, more structural version of this same stack is covered in the schema stack that makes a page citable, which is a far higher-leverage place to spend engineering time than a manifest file with a 97% zero-request rate. Explore more in Lyxel&Flamingo’s SEO and GEO resources.

Budget and engineering time are finite. Every hour spent debating an llms.txt spec update is an hour not spent cleaning up inconsistent product schema across marketplace listings, the company’s own site, and press mentions, work with a far clearer line to AI citation.

Should You Still Publish One?

Publishing an llms.txt file is not harmful. It is cheap, it does not conflict with other GEO work, and if the counter-evidence eventually strengthens, having one in place costs nothing retroactively. The mistake is treating it as a priority investment or a citation strategy on its own.

Teams should publish it if it is trivial to do, then move on immediately to the work the evidence actually supports. That next decision increasingly is not whether to publish a manifest file at all, it is whether to allow, block, or charge AI crawlers for access in the first place, a question Cloudflare’s own default-blocking move has already forced onto a lot of roadmaps whether or not a brand was ready for it. Measuring any of this against a baseline also depends on not undercounting the citations a brand is already earning, the specific gap covered in building a brand metric that doesn’t depend on clicks.

The GEO Priority Framework

Framework Explained

Rank by evidence, not novelty: A newer tactic is not automatically a better one; llms.txt is newer than schema markup and has weaker evidence behind it.

Cost and payoff are separate axes: Low effort does not justify a high priority ranking if the payoff is unproven.

Revisit periodically: AI crawler behavior is still evolving, and a tactic with weak evidence today can strengthen; this table is a snapshot, not a permanent verdict.

Publish, then deprioritize: There is no reason to avoid llms.txt, only a reason to stop treating it as a strategy.

GEO READINESS NOTE: This is a market-wide evidence question rather than a client-specific result, so there is no Lyxel&Flamingo case study to cite here. What the agency’s GEO engagements do consistently prioritize, based on client outcomes, is structured data and answer-first content restructuring over speculative technical files like llms.txt.

Key Takeaways

• A 137,210-domain, server-log-based study found 97% of published llms.txt files received zero AI-crawler requests.
• AI bots did not proactively look for llms.txt files on sites that lacked one, undercutting its core premise.
• A smaller counter-study claims a positive lift. The evidence is far thinner than the large-scale log data.
• Schema markup, entity consistency, and answer-first content structure all have stronger evidence behind them and should be prioritized first.
• Publishing llms.txt is low-risk and low-cost. It is not a citation strategy on its own.

CXO Takeaway

Is your team debating an llms.txt rollout when the bigger citation gaps are still in your schema and content structure? That is the question worth asking before the next GEO sprint gets planned. If your AI-search roadmap needs a second opinion grounded in evidence rather than trend-chasing, Lyxel&Flamingo’s SEO team can help you rank the work that actually moves citations.