llms.txt is a plain Markdown file served at the root of a domain, intended to tell language models what a site is and which pages matter. It was proposed in 2024 and published as a specification at llmstxt.org. It is not enforced by anything: no crawler is required to read it, and none is penalised for ignoring it.
That last sentence is the one most articles about llms.txt leave out. It shapes everything else here. We pulled the file from 229 AI agency domains and read what is actually inside — not what the spec says should be inside.
The full dataset, including every domain and its raw snippet, is in the AI Agency Crawler Policy Study 2026.
What we looked at
Across 229 collected domains, 224 produced a usable robots.txt answer and are the base for our percentages. For each one we requested /llms.txt with a ten-second timeout and a research user agent, and stored the first 1,000 characters of the response.
That storage detail matters for two reasons. First, it lets us check content rather than status codes. Second, it lets us catch a specific failure mode: 10 domains returned HTTP 200 with an HTML page instead of a text file. A status-code-only crawl would have counted those as having an llms.txt. They do not. Once you exclude them, the real number is 144 domains — 64.3% of the sample.
That figure is high, and the next section explains why it should not be read as enthusiasm.
The numbers
- 64.3% serve a real
llms.txt(144 of 224) - 4.5% return 200 with an HTML page — a soft 404 wearing a success code (10)
- 12.1% serve an
ai.txt(27) - 15.3% of the real files carry an SEO-plugin signature in the opening lines (22 of 144)
Those are the headline numbers. The useful part is what the files contain.
What the spec actually asks for
The proposal is deliberately small. Only one element is required: an H1 containing the name of the project or site. Everything else is recommendation — a blockquote summary directly beneath the title, H2 sections grouping links as [title](url): note, and an optional ## Optional section for material that can be dropped when context is limited.
A companion file, llms-full.txt, is described for inlining complete page text rather than summaries. None of it is enforced. There is no versioning, no freshness signal, no way for a publisher to declare "this file is current", and no mechanism by which a site can learn whether anything read it.
That minimalism explains a lot of what we found. When the only hard requirement is a title, a title is what many sites ship. The specification trusts publishers to do the useful part voluntarily, and the data suggests most do not.
What is actually inside
We measured structure across all 144 files. The results are less uniform than the adoption figure suggests:
- 107 begin with an H1 title (
# Something) - 117 contain at least one H2 section
- 113 contain at least one URL
- 1 is under 250 characters — almost all files have some body
Three patterns account for nearly everything we read.
Pattern 1 — The prose summary
A title, a short statement of what the business does, and little else. aisearchready.ca opens with # AI Search Ready followed by a section titled ## What the business does, and contains no URLs at all. anml.com follows a similar shape with ## About and ## Recognition.
These files read well to a human and give a model almost nothing to act on: no page inventory, no canonical entry points, no indication of which page answers which question.
Pattern 2 — The sitemap in prose
The most useful pattern, and the closest to the spec. aiveda.io is organised as ## Pages and ## Posts with links. aktwins.com uses service sections under a named heading. ancient.global includes a language-specific section (## Sitio en español), which is the only file in our sample that visibly handled a multilingual site.
These are the files that could plausibly help an agent navigate a site. They are a minority.
Pattern 3 — The title and nothing else
apptunix.com and appverticals.com both open with a single H1 and essentially stop. A file that says only the company name satisfies the letter of the spec — only an H1 is required — and serves no purpose at all.
Plugin-generated versus hand-written
We tested the opening 1,000 characters for five common plugin signatures: Rank Math, Yoast, All in One SEO, AIOSEO and SEOPress. Twenty-two files matched, overwhelmingly Rank Math, whose output is explicit about its origin: a line beginning "Generated by Rank Math SEO, this is an llms.txt file designed to help LLMs better understand and index this website."
Two caveats, both important.
The true share is probably higher. Five signatures is a small net. A CMS or theme that writes the file without naming itself would not be caught. The 122 files we did not flag are not proven hand-written — they are simply unflagged.
This is not a criticism of the plugins. A generated file is better than no file, and the plugin output we inspected was structurally sound: title, sitemap link, posts. The point is about what the adoption number means. A green tick in a dashboard is not the same as a decision, and 64.3% adoption is measuring installation, not intent.
Five mistakes we kept seeing
Reading 144 files produces patterns. These are the five that recurred most often, roughly in order of how much value they destroy.
- Title-only files. A company name and nothing else. Technically compliant, functionally empty. If an agent learns one fact from your file, the file has done its job.
- Describing the template instead of the business. Generated summaries tend to produce sentences like "we provide innovative solutions to clients worldwide" — true of ten thousand companies and useful for none.
- A bare list of URLs. A sitemap with a different filename. Without a one-line note on what each page answers, the agent has to fetch every page to find out, which defeats the purpose of a summary.
- Links that no longer resolve. We could not measure this systematically from 1,000-character snippets, but we saw plenty of files referencing paths that look like an old site structure. A confident file pointing at dead pages is worse than silence.
- No date and no owner. Nothing in the sample told a reader when the file was last reviewed or who maintains it. A single line — "last reviewed 2026-10-25" — would make the file auditable.
Does any of this work?
This is where we have to be careful, because the honest answer is not commercially convenient.
Google has stated that it does not use llms.txt, and has added guidance describing the file as not required. Studies published during 2026 have looked for a correlation between having the file and being cited by AI assistants, and have not found a measurable one. Log-based analyses have found that the overwhelming majority of requests for the file come from SEO audit tools checking whether it exists — a market observing its own artefact.
For a balanced industry summary of the sceptical position, see CMSWire: Is llms.txt worth your marketing team's time?
So what is it good for? Two things, both modest.
Agent navigation. Coding and browsing agents increasingly fetch clean, curated documentation. If your site is developer-facing, a well-written file is a genuine convenience. If your site is a five-page agency brochure, it is mostly decoration.
Information architecture. Writing one forces you to answer a useful question: which twelve pages represent this business? That exercise has value even if no model ever reads the output.
What it is not is a ranking lever or a citation hack. Treat anyone selling it as either with suspicion. The underlying drivers of AI visibility — being crawlable, structured, attributable and worth citing — are unchanged and are covered in our AEO and GEO practice.
What a good llms.txt looks like
If you are going to publish one, publish one that is worth reading. The structure below follows the specification and borrows the parts of the "sitemap in prose" pattern that we saw working in the sample:
# Company Name
> One or two sentences describing what the company does,
> who it serves, and where it operates.
## Services
- [Service page](https://example.com/service): what it covers, in one line.
## Resources
- [Guide](https://example.com/guide): what question it answers.
## Optional
- [Legal](https://example.com/privacy): not essential for answering questions.Four rules that follow from what we read:
- Write the summary yourself. The blockquote is the only place a model learns what you actually do. Generated summaries describe a template, not a business.
- Link only what matters. Twelve to twenty entries. Every entry you add dilutes the ones that matter.
- Say what each link answers. A bare list of URLs is a sitemap; a one-line note is a decision about relevance.
- Keep it current. A stale file that links to removed pages is worse than no file, because it actively misdirects.
How to check whether assistants cite you
There is no dashboard for this, and anyone claiming otherwise is selling something. What works is deliberate, repeatable questioning.
Build a prompt set. Ten to twenty questions your buyers would actually ask, phrased without your brand name: "best AI web development agencies in Europe", "who can implement answer engine optimization for a B2B SaaS". Run them monthly, in the same tools, and record whether you appear, in what position, and with what description.
Ask the brand question separately. "What does [your company] do?" reveals what the model has absorbed about you — often more useful than whether it recommends you, because it exposes errors in your entity representation.
Watch the referrers. AI assistants increasingly send traffic with identifiable referrers or user agents. A simple log filter will tell you whether cited traffic is arriving at all, which is a better signal than any proxy metric.
Accept the noise. Answers vary between runs, between accounts, and between model versions. A single check proves nothing; a monthly series does.
Run the same prompt set against three or four competitors. A brand appearing in isolation is interesting; a brand appearing while a named competitor does not is a pattern worth acting on. This is also the cheapest form of competitive research available, because it needs no tooling — a spreadsheet and twenty minutes a month.
One structural limitation to keep in mind: assistants are increasingly grounded in search results rather than in memorised training data. That means the ranking work you already do still feeds the answer, and an llms.txt does not compensate for a page that search does not surface. If the underlying pages do not rank, the file describing them will not be read by whatever fetches the results.
If you are building an assistant on your own site, the same discipline applies to how it retrieves and cites sources — the subject of our chatbot work.
Conclusion
64.3% of AI agencies serve an llms.txt, but the number is inflated by plugins, and the files themselves split into three uneven groups: prose summaries that give a model nothing to navigate, sitemap-style files that do, and titles with no body at all.
The file is cheap and harmless. It is also not a strategy. Publish one if you can write it properly and keep it current; skip it if the alternative is a generated stub that describes a template instead of your business. Either way, the decision is worth more than the artefact — and it belongs next to your robots.txt policy, not instead of it.
There is a version of this argument that runs the other way, and it deserves a fair hearing: if a future crawler does read the file, being early costs almost nothing. That is true, and it is the strongest case for publishing one. It is not a case for claiming a benefit that has not been demonstrated. The honest position is that an llms.txt is a cheap bet on a plausible future, made alongside the work that already pays today — crawlable pages, clear entities, and content worth citing.
Our own site publishes one. We wrote it by hand, we link twelve pages rather than a hundred, and we would not claim it has moved a single answer. That is the standard we would apply to anyone else's.
Sources: llmstxt.org: the llms.txt proposal; OpenAI: overview of GPTBot; CMSWire: Is llms.txt worth your marketing team's time?.
Every domain in the sample, with its raw llms.txt opening, is published in the study so you can check our reading against the source.