Knowledge base / Structured data
llms.txt: what it is, what is proven, and why we publish one anyway.
A small file at the root of your site that tells AI tools what matters. Real adoption, genuine uses, and no evidence at all that it affects search. All three are true at once.
Key takeaways
- What is llms.txt? A markdown file at the root of a website listing its most useful pages with short descriptions, so an AI tool can read a clean index instead of parsing navigation and ads.
- Who proposed it and when? Jeremy Howard published the proposal in September 2024, with a second version in 2026. It is a community convention, not a standard from any search engine or standards body.
- Does Google use it? No. Google has said plainly that it does not use llms.txt for Search, and no major search engine documents reading it as a ranking or citation signal.
- So why publish one? Because it takes ten minutes, costs nothing, breaks nothing, and some developer-facing tools do read it. We publish one and label it an emerging convention rather than selling it as a ranking tactic.
What is in the file?
An llms.txt is plain markdown at the root of your domain. It opens with your business name, adds a sentence or two of context, then lists your most useful pages as links with a short description on each. That is the whole format.
The reasoning behind it is sound. An AI tool reading your site has to work through navigation, cookie banners, scripts and footers to find the content. A single clean index of what matters saves it the trouble and reduces the chance it describes you from your own menu labels.
What is actually true about adoption?
This is the part where most writing on the subject picks a side and overstates it. Both of these are true at the same time.
There is real adoption. The proposal was published by Jeremy Howard in September 2024 and revised in 2026. Documentation platforms now generate the file automatically, several AI companies publish one for their own developer docs, and thousands of sites have one. That is a genuine ecosystem in developer tooling.
No search engine has said it uses yours. Google has stated it does not use llms.txt for Search. Neither Microsoft nor the assistant vendors document consuming it when choosing sources for an answer. Every claim that publishing one improves your AI visibility is, at the time of writing, an assertion without published evidence behind it.
Those two facts are compatible. A convention can be widely published and not yet consumed by the systems you care about.
So why do we publish one?
Three reasons, none of them a ranking claim.
It costs about ten minutes and it cannot hurt a site that keeps it current. It is read today by developer tooling, which matters if your buyers are technical. And if any engine does start consuming it, the file is already there and already accurate.
What we will not do is put it on an invoice as an AI visibility service. It is one line in a build, included and unpriced, and it sits last on our eight-point list because the evidence for it is the thinnest. The seven above it have far better reasons behind them.
How to spot the overselling
The file has become a convenient thing to charge for, because it is easy to produce, easy to show the client, and impossible for them to disprove. Three claims to push back on.
- “llms.txt gets you into ChatGPT.” ChatGPT’s web answers come from search results. Get indexed in Bing first, and judge any proposal that skips that step accordingly.
- “It is the new robots.txt.” robots.txt is obeyed by the major crawlers and controls access. This file controls nothing and obliges nobody.
- “It is a ranking signal.” No engine has published anything supporting this. If someone shows you a case study, ask what else changed on the site in the same period. Usually quite a lot.
We would rather lose the sale than make a claim we cannot support. That is the same reason our free score marks anything it could not measure as not measured instead of guessing at it.
Questions about llms.txt
No, and the similar name causes real confusion. robots.txt controls access: it tells crawlers what they may and may not fetch, and the major crawlers obey it. llms.txt controls nothing. It is a suggested reading list, and nothing is obliged to look at it.
There is no evidence that it does, and we would rather say so than take money for it. ChatGPT's web answers come from search results. Being indexed in Bing and having a page that answers the question are what decide that. The file is not a shortcut past either.
Documentation tooling, some coding assistants, and a growing set of developer platforms that generate the file automatically. The AI labs publish them for their own developer docs. That is genuine adoption in one niche, and it is not the same as search engines consuming yours.
Only in two ways, both self-inflicted. If you list pages you did not want surfaced, you have advertised them. And if you publish it once and never update it, you are handing out a stale index of your own site. Treat it as a file that needs maintaining or do not publish it.
A heading with your business name, a one-line summary, then a short list of your important pages as markdown links with a sentence each. Our free llms.txt Generator builds one from a form in your browser. Upload it to the root of your site so it sits at yourdomain.com/llms.txt.
Yes, on every build and uplift, and it is one of the eight points the free score reports. It is also the one point on that list we tell you to care least about, which is the opposite of how it tends to get sold.
See whether your site has one, and the seven things that matter more.
The free AI Visibility Score checks for llms.txt alongside the seven points with better evidence behind them, and tells you which to fix first.
Get your AI Visibility Score