What Is llms.txt — and Does It Actually Matter?

llms.txt is a proposed plain-text manifest, served at /llms.txt, that lists a site's key pages with short descriptions in a format intended to be easy for a language model to parse. The idea, formalized at llmstxt.org, is that a website can hand an AI system a curated table of contents instead of making it guess which pages matter from a full crawl or a sprawling sitemap.

That's the whole concept. It's a markdown file. There's no required schema beyond a loose convention, no submission process, and no verification step. You write it, you publish it, and it sits there.

Why this affects AI citations — with an honest caveat

Here's the part worth being straight about: no major AI engine has publicly committed to reading or prioritizing llms.txt when deciding what to cite. It is a community-proposed convention, not a standard adopted by OpenAI, Anthropic, Google, or Perplexity. If you're looking for a lever that reliably moves citation behavior, this isn't confirmed to be one.

What llms.txt plausibly does is reduce ambiguity for any system — AI-related or not — that chooses to fetch it. If a retrieval system does look for this file, a well-written one hands it a shortcut: instead of inferring your most important pages from crawl depth or internal link structure, the system gets a direct, described list. Whether any given engine takes that shortcut today is unknown and likely varies over time as adoption changes.

This guide is deliberately tip-severity, not issue-severity, for that reason. Missing llms.txt is not the same category of problem as a blocked crawler or a missing meta description — those affect things engines demonstrably do (fetch pages, generate snippets). llms.txt affects something engines might do. It is cheap insurance, not a ranking lever, and you should not expect a citation bump from adding one.

Is llms.txt a standard yet?

No. It's a convention proposed by one project (documented at llmstxt.org) that some sites have adopted voluntarily. There is no governing body enforcing a schema the way there is for sitemaps (which have a formal XML spec) or robots.txt (which has decades of de facto standardization across crawlers). Calling it "the llms.txt standard" overstates where it currently sits — it's closer to an emerging convention that may or may not gain the kind of broad adoption that turns a convention into a standard.

What a good llms.txt contains

The useful structure, per the llmstxt.org proposal, is simple:

1. An H1 with your site or product name.
2. A one-line blockquote summary of what the site is.
3. Optional short paragraphs with additional context.
4. One or more H2 sections, each grouping a set of markdown links with brief descriptions — typically something like "Docs," "Guides," or "Key pages."

The value is entirely in the curation. A file that lists every URL on your site, auto-generated from a sitemap, is redundant with the sitemap and gives an AI system nothing a sitemap didn't already provide. A file that lists the dozen pages you'd hand a new employee on day one — with a sentence explaining why each matters — actually adds information.

Example file

# CitationsAI

> CitationsAI Inspector is a free Chrome extension that extracts
> AI answer citations and audits pages for the factors that earn them.

## Guides

- [AI Crawler Access: robots.txt for GEO](https://citationsai.com/guides/ai-crawler-robots-txt): How GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot access (or don't access) your site.
- [Schema Markup for AI Citations](https://citationsai.com/guides/schema-markup-ai-citations): Which structured data types plausibly help retrieval systems, and which are noise.
- [Writing Citable Definitions](https://citationsai.com/guides/citable-definitions): Why some pages get quoted by AI engines and others don't.

## Product

- [Home](https://citationsai.com/): What the extension does and how to install it.
- [Privacy](https://citationsai.com/privacy): What data the extension does and doesn't collect.

Keep entries to one line each. If a description needs two sentences to explain why a page matters, that's a sign the page itself might need a clearer definition-pattern opening — which is a separate, more consequential fix covered in the citable-definitions guide.

How is llms.txt different from a sitemap or robots.txt?

The three files sound related and sit at similarly predictable URLs, but they do different jobs and none of them substitutes for another. A sitemap (/sitemap.xml) is an exhaustive, machine-generated list of every URL on your site, formatted as XML, intended to help search crawlers discover pages they might otherwise miss — completeness is the point, not curation. robots.txt is a permission file: it tells crawlers what they're allowed or disallowed from fetching, and says nothing about which pages are actually worth reading. llms.txt is neither exhaustive nor a permission mechanism — it's a short, hand-picked, described subset of your content, closer in spirit to a "start here" page than to either of the other two files. Having a sitemap and a correct robots.txt doesn't make llms.txt redundant, because neither of those files curates or describes anything; it also doesn't make llms.txt a replacement for either, since removing your sitemap or robots.txt in favor of it would remove functionality that actual crawlers do rely on today.

How to check

Fetch https://yourdomain.com/llms.txt directly in a browser or with curl. If it 404s, you don't have one. There's no crawler-behavior check to run here the way there is for robots.txt, because no engine publishes confirmation that it fetched the file — this is a presence check, not a behavior check.

How to implement it in 10 minutes

1. List your 10–20 most important pages — the ones you'd want a new team member, a partner, or an AI system to read first.
2. Write one sentence per page describing what it covers and why it's useful, not just restating the title.
3. Group them under two or three H2 headings that make sense for your site (Guides, Product, Docs — whatever fits).
4. Add the one-line site summary at the top.
5. Save it as a static file at /llms.txt and deploy.
6. If your site framework generates content from structured data (like MDX frontmatter), consider generating this file at build time from the same source, so it can't silently drift out of sync with your actual page list.

That's the entire process. There's no submission, no verification service, no review queue — it's a static file like robots.txt or a sitemap.

When to skip it

If you're deciding where to spend a limited amount of GEO time, llms.txt should be near the bottom of the list, not the top. A blocked AI crawler, a missing meta description, or an opening paragraph that never states a clear definition will affect citation behavior in ways that are mechanically explainable — the crawler can't fetch the page, the engine has no summary to work from, or there's no extractable claim to quote. Those are worth fixing first.

Do it once you've handled the pages engines can already demonstrably read and summarize, and treat it as what it is: a low-cost, non-load-bearing addition that costs you ten minutes and asks for nothing in return except the possibility, not the promise, that some future or present system finds it useful.

What happens if adoption never grows?

Worth planning for honestly: it's possible that llms.txt stays a niche convention adopted by a subset of security-conscious or GEO-conscious sites, with no engine ever formally committing to read it, the same way a number of proposed web conventions have quietly stalled at "some sites use this" without becoming universal. If that happens, the ten minutes you spent were still ten minutes, not a wasted strategy — the file costs nothing to maintain once written, doesn't interfere with anything else on your site, and carries no downside if it turns out nobody's reading it. That asymmetry — low cost, uncertain but nonzero upside, no real downside — is the entire honest case for doing it, and it's also exactly why this guide won't tell you it's more important than it is. Spend your real effort on the fixes with a demonstrable mechanism, and let this one sit quietly in the background as exactly the kind of cheap, unglamorous housekeeping task it is.