Last reviewed:
What is the llms.txt file? Definition and business implications
llms.txt is a proposed web standard introduced in September 2024 by Answer.AI, providing AI models with a structured map of a website's content. Placed at the root of the domain, it is not a formal standard and its adoption by major LLMs remains uneven in 2026.
The llms.txt format is inspired by robots.txt but plays an inverse role: instead of forbidding access to certain parts, it orients AI models towards content deemed strategic. It lists, in markdown, the URLs of main resources with a short description of each. Adoption in 2026 is uneven. On the publisher side: Anthropic, Stripe, Vercel, Cloudflare, Cursor, Hugging Face publish their llms.txt. Yoast SEO has integrated it into its WordPress plugin. On the consumer side: developer tools (Cursor, Continue, Aider) actively read the file; consumer LLMs (ChatGPT, Claude.ai, Perplexity) do not formally commit, even if their crawlers sometimes fetch it. A SERanking study from November 2025 on 300,000 domains found no statistical correlation between the presence of a llms.txt and improved citations in AI answer engines.
Concrete example
Anthropic publishes its llms.txt on docs.anthropic.com, with a standard structure: an introduction describing the documentation, then sections (Get started, API reference, Use cases) listing each important page with a one-sentence description. The file is complemented by an llms-full.txt containing the full text of the documents for direct ingestion. Pattern reproduced by Vercel, LangGraph, and the majority of developer-facing companies in 2026. For a mid-cap without API documentation, the stake is different: a useful llms.txt typically lists 20 to 50 URLs (product pages, glossary, case studies, terms and conditions) rather than an exhaustive technical library. Production cost: between 4 hours for a simple site and 2 days for a complex site.
See also
Further reading
Sources
- llms.txt proposal, maintained at llmstxt.org, introduced by Jeremy Howard, Answer.AI, September 2024. https://llmstxt.org/
- Anthropic documentation, llms.txt implementation example. https://platform.claude.com/llms.txt