What is llms.txt? Format, examples and whether you need one

llms.txt is a markdown file, usually served at the root of a website as /llms.txt, that gives large language models and AI agents a concise overview of the site and links to the pages most worth reading. It was proposed by Jeremy Howard of Answer.AI on September 3, 2024, and the proposal at llmstxt.org was revised to version 2 in August 2026.

The problem it addresses is practical. Web pages are built for people, wrapped in navigation, scripts and layout, and converting them back to clean text is imprecise. Context windows are still too small to read most sites whole, and every wasted token costs time and money. An llms.txt file gives an agent a short, expert-written starting point: what this site is, how to interpret it, and where the detail lives.

It is a proposal, not a ratified web standard, and adoption is uneven. The proposal itself says llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials. Business sites can use the same structure, but with lower and less certain returns.

The format is plain markdown in a fixed order, so a model can read it and ordinary code can parse it. The file can also live below the root, such as /docs/llms.txt, in which case it covers only the pages under that path, and agents are meant to use the most specific file that applies.

The proposal also recommends offering clean markdown versions of important pages at the same URL with .md added, and announcing them with an alternate link of type text/markdown in the page head or an HTTP Link header. The sections of the file itself, in order, are:

• An H1 with the name of the project or site. This is the only required section.
• A blockquote with a short summary containing the key information needed to understand the rest of the file.
• Optional paragraphs or lists, with no headings, that explain the project and how to interpret the linked pages.
• Zero or more sections under H2 headings, each a markdown list of links, with an optional colon and a one-line note after each link.
• An optional section headed Optional, for secondary links an agent can skip when it needs a shorter context.

llms-full.txt is a companion convention, not part of the llmstxt.org proposal. Where llms.txt is a short index of links, llms-full.txt puts the full text of the important content into one file, so an agent can load everything in a single request instead of following links. Documentation platforms popularized it. Mintlify, for example, generates both files automatically and describes llms-full.txt as combining an entire documentation site into a single file.

The tradeoff is size. A full-text file for a large site can exceed what many context windows hold, which defeats the purpose. It works best for focused sites and documentation sets. If you publish one, generate it from the same source as your pages and leave out boilerplate such as navigation, legal text and repeated calls to action.

Google is explicit. Its guide to optimizing for generative AI search says you don't need AI text files, markup or markdown to appear in Google Search, including AI Overviews and AI Mode, because Google Search itself doesn't use them. It adds that keeping an llms.txt file for other services is fine and will neither harm nor help your visibility or rankings. Google added that clarification to its documentation in June 2026.

The readers that exist are agents working on demand: coding assistants fetching library documentation, and chat assistants or browser agents that look for a site's llms.txt while helping a user. The proposal notes that documentation platforms generate the file automatically, that Chrome's Lighthouse checks for one as part of its agentic browsing audits, and that OpenAI, Anthropic and Google's Gemini team publish llms.txt files for their own developer docs. Publishing a file is not the same as reading other sites' files, though.

As of October 2026, the crawler documentation from OpenAI, Anthropic and Perplexity that we checked does not say their search crawlers use llms.txt for indexing or ranking. Treat any claim that llms.txt boosts AI citations as unproven unless the vendor can point to documentation that says so.

The three files sit side by side at the root of a site and are often confused, but they answer different questions. robots.txt says which automated clients may crawl which paths; it is about permission. sitemap.xml lists every indexable URL so search engines can discover and recrawl them; it is about completeness. llms.txt is a short, human-written guide to the pages an agent should read first; it is about curation.

None replaces the others. llms.txt does not block or allow anything, so it cannot keep content out of training or search; that is robots.txt's job, per user agent. A sitemap is no substitute for llms.txt either. As the proposal points out, a sitemap usually lacks LLM-readable versions of pages, cannot point to useful external resources, and lists far more than fits in a context window. The table on this page compares all three, plus llms-full.txt and structured data.

A useful llms.txt takes an hour or two for a small site. The hard part is deciding what to leave out, because a file that lists every URL is just a worse sitemap.

1. Pick the scope: one file at the root for the whole site, or separate files under paths such as /docs/ if sections serve different audiences.
2. Write the H1 and summary. Use your exact business or product name, then a one- or two-sentence blockquote saying what you do and for whom.
3. Add interpretation notes: short paragraphs with facts an agent might get wrong, such as services you do not offer, regions you serve, naming conventions or which version is current.
4. Choose your key pages, usually a few dozen at most. Group them under H2 headings such as Services, Pricing, Guides or Documentation, and give each link a one-line note saying what the page answers.
5. Move secondary links into an Optional section so agents with tight context limits can skip them.
6. Link to clean content. Where you can, point to markdown versions of pages, or at least to pages whose main content is in the server-rendered HTML.
7. Serve it as plain text at /llms.txt with a 200 status, and check that robots.txt and bot protection do not block it.
8. Generate it instead of hand-maintaining it. Build the file from your CMS or content data at deploy time so new and retired pages are reflected automatically.
9. Test it. Give an assistant only your llms.txt, ask it questions about your business, and fix whatever it gets wrong.

Our own site publishes both /llms.txt and /llms-full.txt, and neither is written by hand. Both are generated at build time from the same content registries that render the website: services, use cases, guides, case studies and field notes. When we add or retire a guide, both files change with the next deploy, so they cannot drift from the pages.

The short file follows the proposal's shape. It opens with an H1 for the company name and a blockquote summary, then H2 sections for what we do, services grouped by category, our process, use cases, guides, case studies and field notes. Each entry is a link with a one-line description taken from that page's own metadata. The full file expands each service with what is included, how the work runs and examples, so an agent can answer detailed questions without visiting every page.

We pair it with a robots.txt that explicitly allows OpenAI's and Anthropic's search, user-request and training crawlers. We treat all of this as low-cost context for agents. We do not expect it to move Google rankings, and Google has said it will not.

If you run on a common platform, you may not need to build anything. The llms.txt proposal lists several tools that generate the file automatically: Mintlify, which also creates markdown versions of every docs page; GitBook, for published docs sites; the Yoast SEO and AIOSEO plugins for WordPress; and Wix, which generates one for every Wix site. Several of the same tools handle structured data too. Yoast SEO and AIOSEO output schema.org markup for WordPress pages, Wix offers structured data settings, and many Shopify themes include basic product markup.

Custom sites built on frameworks such as Next.js, Astro or Nuxt usually generate both from code: a route that renders llms.txt from the content source, and JSON-LD components fed by the same data as the page. That takes more work up front but keeps the file, the markup and the visible page consistent.

Whichever route you take, check the output by hand. Auto-generated files often list every URL, including thin tag and archive pages, which defeats the point of a curated guide. And no platform's llms.txt or schema output will fix pages that do not answer the questions your buyers ask.

It is low cost and low certainty. For most business websites an llms.txt file takes a few hours once, and nothing after that if it is generated. In return, agents that look for it get a cleaner starting point. What you should not expect is a ranking boost in Google, or proof that ChatGPT, Perplexity or Claude cite you more because of it.

Prioritize it if you publish developer documentation, run a SaaS product with an API, or expect customers to send AI agents to your site to compare plans or complete tasks. Deprioritize it if your basics are not done: crawlable pages, server-rendered content, accurate entity facts and pages that answer real questions. Those affect every AI engine. llms.txt, at best, helps some of them.