Last reviewed:
What is the llms.txt file? Definition and business implications
llms.txt is a proposed web standard introduced in September 2024 by Answer.AI, providing AI models with a structured map of a website's content. Placed at the root of the domain, it is not a formal standard and its adoption by major LLMs remains uneven in 2026.
The llms.txt format is inspired by robots.txt but plays an inverse role: instead of forbidding access to certain parts, it orients AI models towards content deemed strategic. It lists, in markdown, the URLs of main resources with a short description of each. Adoption in 2026 is uneven. On the publisher side: Anthropic, Stripe, Vercel, Cloudflare, Cursor, Hugging Face publish their llms.txt. Yoast SEO has integrated it into its WordPress plugin. On the consumer side: some developer tools, such as Cursor through its @Docs feature, can index it; consumer LLMs (ChatGPT, Claude.ai, Perplexity) do not formally commit, even if their crawlers sometimes fetch it. In June 2026, Google clarified that the file is not needed for its search engine and influences neither visibility nor ranking in Google Search. A SERanking study from November 2025 on 300,000 domains found no statistical correlation between the presence of a llms.txt and improved citations in AI answer engines.
Concrete example
Anthropic publishes its llms.txt on platform.claude.com (the former docs.anthropic.com address redirects there), with a standard structure: an introduction describing the documentation, then thematic sections (models and pricing, API reference, SDKs, release notes) listing each important page, often with a short description. The file is complemented by an llms-full.txt containing the full text of the documents for direct ingestion. Pattern reproduced notably by Vercel, LangChain and Cloudflare. For a mid-cap without API documentation, the stake is different: a useful llms.txt lists the strategic URLs (product pages, glossary, case studies, terms and conditions) rather than an exhaustive technical library.
FAQ
What is the llms.txt file?
It is a proposed web standard introduced in September 2024 by Answer.AI that gives AI models a structured map of a site's content. Placed at the root of the domain, it lists in markdown the URLs of the main resources with a short description. It is not a formal standard, and its adoption by major LLMs remains uneven in 2026.
How does it differ from robots.txt?
It plays an inverse role. Where robots.txt forbids access to certain parts of the site, llms.txt instead orients AI models towards the content deemed strategic.
Do major LLMs actually read it?
Adoption is uneven. Some developer tools, such as Cursor through its @Docs feature, can index it, but consumer LLMs (ChatGPT, Claude.ai, Perplexity) do not formally commit to it. In June 2026, Google clarified that it is not needed for its search engine and influences neither visibility nor ranking. A SERanking study from November 2025 on 300,000 domains found no statistical correlation between its presence and improved citations.
Should we deploy one?
The real question is: do your users query AI about your offer (technical documentation, API, product sheet)? If so, llms.txt has immediate value. For others, it is an option, not a priority.
What does it cost and what can we expect?
Deploying it takes little work, as it is a plain text file: the cost argument carries little weight. But without a baseline measured before deployment (citations in Perplexity, AI Overviews), you will not know whether it paid off. The WYP rule: deploy, measure, do not oversell.
See also
Further reading
Sources
- llms.txt proposal, maintained at llmstxt.org, introduced by Jeremy Howard, Answer.AI, September 2024. https://llmstxt.org/
- Anthropic documentation, llms.txt implementation example. https://platform.claude.com/llms.txt
- Google's Guide to Optimizing for Generative AI Features on Google Search, Google Search Central, 2026. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- Google Search Central, documentation updates log: FAQ rich result discontinued (7 May 2026) and llms.txt note (June 2026). https://developers.google.com/search/updates