If your AI search checklist starts and ends with “add an llms.txt file,” it’s time for a new checklist. In May 2026, Ahrefs pulled server-log data from 137,210 domains and found that 97% of published llms.txt files received zero requests that month (Ahrefs, 2026). Of the traffic the remaining 3% did get, most of it came from SEO audit tools and general crawlers, not from ChatGPT, Claude, or Perplexity actually reading the file before answering a question. Google has been direct about it too: at a Search Central Deep Dive, Gary Illyes said flatly that Google doesn’t support llms.txt and isn’t planning to, and John Mueller has compared it to the keywords meta tag, a field site owners control that search engines learned to ignore (Search Engine Land, 2026).
None of that means large language model optimization (LLMO) is a dead end. It means the file everyone rushed to publish in 2024 and 2025 was never the leverage point. For a marketing or engineering team at a Series A–IPO SaaS, fintech, cybersecurity, manufacturing, or e-commerce company, the actual work of getting represented accurately in AI-generated answers happens somewhere else entirely: in what your servers let AI crawlers see, in how consistently your brand’s facts are declared across the web, and in whether anyone on your team is actually watching the logs. This piece walks through a five-layer framework for doing that work in the order it pays off, with an honest ranking of where llms.txt actually belongs on the list.
LLMO Is Not GEO, AEO, or Traditional SEO, and That Distinction Matters Here
We’ve written before about how AEO, GEO, SEO, and LLMO differ, and the short version matters for this article specifically: GEO is about earning a live citation inside a generated response (ChatGPT or Perplexity retrieving and quoting your page in real time). AEO is about capturing a direct answer box or featured snippet. LLMO is different from both. It’s about how a model represents your brand from what it absorbed during training and from the entity data it can cross-reference, independent of whether it fetched your page in that exact session. That’s why the LLMO toolkit looks less like “write more citable content” and more like “make sure your crawl access, your structured facts, and your brand’s footprint across the web all say the same thing.”
What the Data Actually Shows
Three data points should shape how any technical marketing team spends its next quarter on this:
- llms.txt has essentially no adoption payoff right now. Ahrefs’ study found AI training crawlers accounted for just 5.3% of the requests that did hit llms.txt files, with AI assistants and retrieval bots even lower. The file exists on 28% of the domains they studied, but almost nothing reads it (Ahrefs, 2026).
- Google has told the market to stop treating it as a ranking or visibility lever. Its stated position is that normal SEO and structured data practices are what surface content in AI Overviews, not a separate LLMO file (Search Engine Land, 2026).
- AI crawlers take far more than they give back, and the ratio varies enormously by vendor. Cloudflare’s own crawl data put Anthropic’s crawl-to-refer ratio at roughly 50,000:1, OpenAI’s at 887:1, and Perplexity’s at 118:1 during the period it measured, meaning some crawlers pull thousands of pages for every visitor they ever send back (Cloudflare, 2025). If your infrastructure team is going to spend engineering time on AI crawler policy, it should go toward deciding which of those bots earns default access, not toward a text file almost nothing reads.
The LLMO Access Stack: A Five-Layer Framework
Once you set aside llms.txt as the centerpiece, the remaining LLMO work sorts cleanly into five layers, ordered by how much verifiable leverage each one gives you. We use this stack with clients to sequence the work instead of treating “AI optimization” as one undifferentiated task.
Explicit robots.txt rules per AI bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) instead of one blanket allow or block.
Organization, Product, and FAQPage schema declaring who you are, what you sell, and to whom, in a format machines parse without guessing.
The same company facts (name, category, founding date, leadership, HQ) repeated identically on Wikidata, Crunchbase, LinkedIn, and G2, linked back with sameAs markup.
Low cost, low current payoff. Worth publishing as a future-proofing hedge and for coding-agent use cases, not as your primary AI visibility lever.
Server log review for actual bot behavior, plus periodic prompting of ChatGPT, Claude, and Perplexity to check how your brand gets described.
Layer 1: Crawler Access Control
Most sites still run a single “allow all” or “block all” rule for every bot with “AI” or “GPT” in its name. That’s a blunt instrument. A fintech company with public pricing pages and gated compliance documentation has a different access calculus than a SaaS company whose entire value proposition is public documentation. Write per-bot rules in robots.txt: allow GPTBot, ClaudeBot, PerplexityBot, and Google-Extended on public marketing and docs paths, and disallow them on gated app routes, account areas, and anything under active legal review. Cybersecurity vendors in particular should treat this deliberately: CISOs are reading your content to evaluate you, and a training crawler that ingests unreleased vulnerability advisories or customer-specific pages is a real exposure, not a hypothetical one.
Layer 2: Structured Entity Data
This is the layer most teams already have partial coverage on through basic SEO work, and it’s worth doing well rather than doing twice. Organization schema should declare your legal name, logo, founding date, and industry. Product or SoftwareApplication schema should state what the product actually does in plain, unambiguous language a model can lift directly. We’ve covered the mechanics of this in more depth in our schema markup guide for AI search; the summary for LLMO purposes is that vague marketing copy (“the modern platform for modern teams”) gives a model nothing concrete to extract, while a specific, structured claim (“SOC 2 Type II compliant expense management software for mid-market finance teams”) gives it something to repeat accurately.
Layer 3: Cross-Web Entity Consistency
Large language models build an internal representation of your company from many sources during training, not just your own site. If your Crunchbase profile lists an old funding round, your LinkedIn page uses a different tagline than your homepage, and Wikidata doesn’t list you at all, a model has conflicting or thin signal to draw from. The practical fix: audit your Crunchbase, LinkedIn company page, G2 or Capterra profile (whichever applies to your category), and Wikidata entry (create one if you qualify under their notability guidelines) for identical company description, founding date, and category language, then add sameAs properties in your Organization schema pointing to each of them. This is slower, less glamorous work than publishing a file, and it’s also the layer with the clearest mechanism for how a model actually learns who you are outside of live retrieval.
Layer 4: Machine-Readable Context (llms.txt)
Publish it if you want a defensible, low-cost hedge, particularly if your product involves developer tooling or documentation that coding agents consume directly, since that’s the one use case where practitioners see llms.txt actually getting read today. Just don’t budget it as your primary AI-visibility initiative, and don’t let it displace the layers above it. Given the adoption data, treat it as a half-day task, not a quarter’s roadmap item.
Layer 5: Verification and Monitoring
None of the first four layers matter if nobody checks whether they’re working. Two habits cover most of what a lean marketing or RevOps team needs. First, pull AI bot user agents out of your server or CDN logs monthly and look at what’s actually being requested: which paths, how often, and whether the crawlers you explicitly allowed are the ones showing up. Second, run a recurring set of prompts against ChatGPT, Claude, and Perplexity asking about your category and your company by name, and log how accurately each one describes you. The mockup below shows the kind of breakdown this monitoring produces once you have a few months of log data to look at.
How the Five Layers Compare
| Layer | Setup Effort | How You Verify It | Priority |
|---|---|---|---|
| 1. Crawler Access Control | Low (hours) | Server logs show the bots you allowed actually crawling; disallowed bots stay out | High |
| 2. Structured Entity Data | Medium (days) | Google Rich Results Test passes; AI answers echo the specific claims you structured | High |
| 3. Cross-Web Entity Consistency | Medium-High (weeks) | Prompt each AI platform for your company facts; check for contradictions across sources | High |
| 4. llms.txt | Low (half a day) | Log requests to the file itself: expect nearly none per current data | Low |
| 5. Verification and Monitoring | Low, ongoing | Monthly log review plus recurring prompt audits across ChatGPT, Claude, Perplexity | High (recurring) |
Where This Fits for Each ICP
The five layers apply everywhere, but the emphasis shifts by vertical. SaaS companies with public documentation get the most immediate return from Layer 1 (deliberately allowing crawlers on docs paths) since that content is exactly what training crawlers and retrieval bots are looking for. Fintech and cybersecurity companies need to spend more time on Layer 1’s exclusions: compliance pages, security advisories, and anything under NDA shouldn’t be sitting behind a blanket “allow all AI bots” rule. Manufacturing and industrial B2B companies often have the weakest Layer 3 coverage, since spec sheets and distributor relationships rarely make it onto Wikidata or get a clean Crunchbase entry, which is exactly why model outputs about industrial brands tend to be thin or outdated. E-commerce brands on Shopify, WooCommerce, or headless stacks should prioritize Product schema under Layer 2, since that’s the structured claim data most directly tied to being described accurately in a shopping-related AI answer.
Frequently Asked Questions
What is llms.txt and should I create one?
llms.txt is a proposed plain-text file, placed at your site root, meant to give AI systems a curated, markdown-formatted summary of your site’s most important pages. You can create one as a low-cost, half-day task, but current data shows it isn’t a meaningful AI-visibility lever: Ahrefs found 97% of published llms.txt files got zero requests in a given month, and Google has said it doesn’t use the file at all (Ahrefs, 2026). Treat it as optional future-proofing, not a priority.
Where does llms.txt go on a site?
At the domain root, the same location as robots.txt, for example yourcompany.com/llms.txt. It should be a plain markdown file, not HTML.
What are llms.txt best practices if I do publish one?
Keep it short: an H1 with your company name, a one-line summary, and a curated list of links to your most important documentation or product pages with brief descriptions. Don’t treat it as a place to stuff keywords or duplicate your entire sitemap. There’s no evidence that volume helps, and it undermines the file’s stated purpose as a curated summary.
What’s the difference between LLMO and GEO?
GEO (Generative Engine Optimization) is about earning a live citation when a model retrieves and quotes your page in response to a specific prompt. LLMO is about how a model represents your brand from training data and cross-referenced entity information, independent of live retrieval. They overlap in practice (structured, factual content helps both) but the mechanisms are different, which is why they call for different tactics.
What’s the difference between LLMO and SEO?
Traditional SEO optimizes for ranking in a search results page a human scans and clicks through. LLMO optimizes for how accurately and favorably a model summarizes or represents your brand inside a generated answer, where there’s no results page or click at all. Many of the underlying inputs (structured data, factual accuracy, entity clarity) support both, but LLMO adds cross-web entity consistency work that classic SEO checklists never covered.
How do I manage AI crawler access without blocking search engines I still need?
Write separate, explicit robots.txt blocks for each bot rather than one wildcard rule. Googlebot (classic search), Google-Extended (Gemini/AI training), GPTBot, ClaudeBot, and PerplexityBot are all distinct user agents and can each get their own allow or disallow rules and even different disallowed paths.
What is entity optimization for AI search?
It’s the practice of making sure your company’s core facts (name, category, founding details, leadership, product claims) are declared consistently in structured data on your own site and repeated identically across third-party sources a model may have trained on, such as Wikidata, Crunchbase, and LinkedIn. Consistent, structured facts give a model a stable source to converge on; contradictory or absent facts leave it guessing or defaulting to outdated information.
Where to Start
If you’re a marketing or growth leader trying to prioritize this work against everything else on the roadmap, start with Layer 1 and Layer 5: fix your crawler access rules this week, and put a recurring log review and prompt-audit process on the calendar next. Those two moves are cheap, verifiable within 30 days, and don’t require a redesign or a developer sprint. Layers 2 and 3 are the higher-effort, higher-payoff work worth scoping properly rather than rushing. If you want a structured read on where your site currently stands across all five layers, our GEO audit maps exactly this: crawler access, structured data coverage, and entity consistency, delivered as an AI citation map within five days.
Share this article
Ryan Brooks leads technical SEO at MV3 Marketing, specializing in schema architecture, entity graphs, crawlability, and the structural signals that determine whether AI answer engines cite a page.
Ready to audit your organic growth opportunity?
$2,500 flat. 5 business days. Six deliverables tied to pipeline , not rankings. No retainer required.
Get the Organic Growth Audit →