Most llms.txt guides skip the uncomfortable part: the file that vendors pitched as an AI-visibility must-have mostly sits unread. A June 2026 Ahrefs analysis of server logs across 137,000 domains found that 97% of published llms.txt files received zero requests in a given month, and that actual AI retrieval bots (the ones fetching pages to answer a live prompt) accounted for just 1.1% of the requests that did happen. The biggest requesters were SEO audit tools checking whether the file existed at all.
That data does not mean llms.txt is worthless. It means most teams are building it for the wrong reason, in the wrong shape, and then never checking whether anything actually reads it. This guide covers what the file is, when it is worth a real investment versus a half-day placeholder, how to build one that follows the actual spec, and how to check your own server logs to see whether anything is requesting it at all.
What llms.txt actually is
llms.txt is a plain-text Markdown file placed at a site’s root (yourcompany.com/llms.txt) that gives AI systems and coding agents a curated index of a site’s most important pages, written in a format a language model can parse without rendering JavaScript or crawling a full sitemap. It is a community convention, not an official standard: there is no IETF or W3C backing, and Google has stated it does not use the file for Search or AI Overviews. The specification itself is maintained at llmstxt.org, and it defines a deliberately small structure: an H1 with the site or project name (the only required element), an optional one-line blockquote summary, an optional explanatory paragraph, and H2 sections listing links with short, specific descriptions.
It is easy to confuse this file with what actually drives AI citations. It is not the same discipline as GEO (earning a live citation when a model retrieves your page for a specific prompt) or the broader entity and structured-data work covered in our LLMO checklist. llms.txt is narrower: it is a curated pointer file, closer to a hand-written table of contents for machines than an optimization tactic.
What the 2026 data actually shows
Three data points worth knowing before you spend real time on this:
- 97% get zero requests. Ahrefs’ log analysis across 137,000 domains found the overwhelming majority of published llms.txt files were never fetched in the study month. Coverage of the same study ran in Search Engine Journal.
- AI retrieval bots are a small share of the requests that do happen. Per the same dataset, live AI-assistant retrieval accounted for roughly 1.1% of llms.txt requests; SEO audit and monitoring tools made up a larger share, meaning most “traffic” to the file is tooling checking for its existence, not a model reading it to answer a question.
- Major AI platforms have not confirmed using it for ranking or citation. Google has said directly it does not use llms.txt. No major AI platform has published a statement that it prioritizes llms.txt content over normal crawling and retrieval.
Read together, this points to llms.txt functioning today as agent-readiness infrastructure, useful for coding agents and internal tooling that explicitly look for it, rather than a lever that moves AI Overview or chatbot citation rates on its own. That is also the conclusion in our own crawler-log research: see AI Crawler Log File Analysis for how to check what GPTBot, ClaudeBot, and PerplexityBot are actually requesting on your own domain.
The llms.txt Investment Framework: three tiers, not one answer
“Should we build an llms.txt file” is not a yes/no question with one correct answer for every company. The right amount of effort depends on what kind of site you run and who is actually consuming your docs programmatically. Use this to decide which tier you are in before you write anything.
Most readers of this blog, B2B SaaS and fintech companies with a marketing site and a docs subdomain, land in Tier 2. Tier 3 is for companies where developers are actually pointing coding agents at your documentation as part of an integration, which is a different, higher-value audience than a chatbot answering a general question about your category.
How to build one correctly (six steps)
- Inventory your must-know pages. Pricing, core product docs, API reference, and two or three pages that answer “what does this company actually do.” Resist the urge to list everything; the file is a curated index, not a sitemap dump.
- Draft to the actual spec. One H1 with your company or project name, one blockquote summary sentence, an optional short paragraph of context, then H2 sections (for example “Docs,” “API,” “Pricing”) each listing links with a specific one-line description. “Full REST endpoint catalog with authentication and rate-limit details” is a usable description; “API docs” is not.
- Keep it plain Markdown, not HTML. The file needs to parse cleanly with a standard Markdown parser. Stray HTML, unescaped brackets, and broken links are the most common validation failures.
- Publish it at the domain root. Same location as robots.txt: yourcompany.com/llms.txt, served as plain text, not behind a redirect or a JavaScript-rendered route.
- Validate it. Run it through a spec-aware checker (see the comparison below) before treating it as done. A file with broken links or malformed Markdown is worse than no file, since it wastes the attention of the tools that do check it.
- Monitor your server logs, not your assumptions. Pull a 30-day window of requests to /llms.txt and compare it against total GPTBot, ClaudeBot, and PerplexityBot activity on the rest of the site. If the file gets close to zero hits after 90 days, that is real information: stop investing further effort in Tier 2, and put that time into the entity and structured-data work that our own crawler-log research shows actually correlates with citation activity.
Checking and generating llms.txt: what the tools actually do
Because llms.txt has no formal schema, “validation” mostly means checking it against the llmstxt.org conventions, not a strict grammar. A handful of free tools do this well enough to catch the common mistakes: broken links, missing descriptions, and malformed Markdown.
| Tool | Generates from sitemap | Validates existing file | Link-health check |
|---|---|---|---|
| Rankability llms.txt tool | Yes | Yes | Basic |
| Chat Thing generator/validator | Yes | Yes | Yes, with a grade |
| llmtxt.info | Form-based scaffold | Yes, with diagnostics | Yes |
None of these replace step six above. A clean validator pass tells you the file is well-formed. It does not tell you anything actually reads it; only your own server logs answer that.
Illustrative pattern from server-log research, not a live client account:
This is the shape the Ahrefs dataset points to at the aggregate level, and it is worth confirming on your own domain rather than assuming it. If your logs look meaningfully different, that is a more useful data point than any generic benchmark.
Frequently Asked Questions
What is llms.txt?
llms.txt is a plain-text Markdown file placed at a website’s root that gives AI systems and coding agents a curated, human-and-machine-readable index of a site’s most important pages. It follows a community specification maintained at llmstxt.org, not an official web standard.
Where does llms.txt go on a website?
At the domain root, the same location as robots.txt: yourcompany.com/llms.txt. It must be served as plain text, not as HTML and not behind a redirect.
Does llms.txt actually help SEO or AI visibility?
Current evidence says no, not directly. Google has stated it does not use llms.txt for Search or AI Overviews, and Ahrefs’ 2026 analysis of 137,000 domains found AI retrieval bots made up only about 1.1% of llms.txt requests, with 97% of published files receiving zero requests in the study month. Treat it as agent-readiness infrastructure, not a ranking or citation tactic.
Should I create an llms.txt file?
For most B2B SaaS, fintech, and cybersecurity marketing sites with a docs section, yes, as a half-day baseline build, because the cost is low even though the current evidence for direct AI-visibility impact is weak. Skip it entirely only if you run a simple marketing site with no docs or API surface. Invest well beyond the baseline only if developers are actively pointing coding agents at your documentation.
What are llms.txt best practices?
Keep it short and curated: one H1 with your company name, one blockquote summary, and H2 sections listing your most important pages with specific, non-generic descriptions. Prioritize canonical pages over duplicates, and do not treat it as a place to dump your full sitemap.
How do I create an llms.txt file?
Write it by hand as plain Markdown following the llmstxt.org structure, or scaffold a first draft with a sitemap-based generator and then edit the descriptions manually, since generated descriptions are usually too generic to be useful on their own.
What’s the difference between llms.txt and robots.txt?
robots.txt controls crawler access, telling bots what they are allowed to request. llms.txt does not control access at all; it is a curated index for bots that choose to read it. A blocked page in robots.txt stays blocked regardless of whether it is also listed in llms.txt.
How do I check or validate an llms.txt file?
Run it through a spec-aware validator (Chat Thing’s and llmtxt.info’s tools both check link health and Markdown structure) to catch broken links and formatting errors. Then check your own server logs for requests to /llms.txt over a 30-day window to see whether anything is actually reading it, which a validator cannot tell you.
If you want the deeper structured-data and entity work that the current evidence shows actually correlates with AI citation activity, not just the llms.txt file, our Claude SEO Agency service covers the full crawler-access and entity-optimization stack for B2B SaaS and fintech teams. For a broader look at where your site stands across AI Overviews, ChatGPT, Perplexity, and Claude citation activity, start with a GEO audit.
Share this article
Ryan Brooks leads technical SEO at MV3 Marketing, specializing in schema architecture, entity graphs, crawlability, and the structural signals that determine whether AI answer engines cite a page.
Ready to audit your organic growth opportunity?
$2,500 flat. 5 business days. Six deliverables tied to pipeline , not rankings. No retainer required.
Get the Organic Growth Audit →