llms.txt, Explained: The Spec, What Actually Reads It, and a Working File for a Security Vendor

llms.txt is a proposed markdown file at your site root that summarises your best content for large language models. As of mid-2026, no major AI company parses it, Google has stated it is not required for Search, and Ahrefs found 97% of llms.txt files received zero requests in May 2026. There is still a narrow case for shipping one, and this is the honest version of that case.

TL;DR

• llms.txt is a proposal, published by Jeremy Howard in September 2024. It is not a standard, and no engine is obliged to read it.
• Originality.ai tracked llms.txt across 3 million sites: 4,088 files in June 2025 grew to 36,120 in May 2026, an 8.8x rise. Adoption is real.
• Ahrefs analysed server logs across 137,000 domains and found 97% of llms.txt files got zero requests in May 2026. Consumption is not.
• Of the 3% that were fetched, SEO audit tools accounted for 21.7% of requests. GPTBot accounted for 4.51%, ClaudeBot 0.80%, and AI retrieval bots in aggregate 1.1%.
• Calling llms.txt "the new robots.txt" is wrong in a way that matters: robots.txt is honoured by every major crawler, llms.txt is honoured by essentially none.
• Ship one anyway if you have real documentation, because the cost is 30 minutes and the option value is real. Do not ship one instead of fixing page structure.

What llms.txt actually specifies

The proposal is small, which is a point in its favour. A file at /llms.txt, written in markdown, structured as:

• An H1 with the project or company name. Required.
• A blockquote with a short summary. Optional but expected.
• Any number of markdown paragraphs or lists giving context.
• H2 sections, each containing a list of links in the form [title](url): notes.
• A final H2 named Optional, whose links may be skipped when the consumer needs a shorter context.

The motivating problem is real. An LLM asked to reason about your product has a fixed context budget and your site is mostly navigation, boilerplate, and marketing prose. A curated markdown index of your genuinely useful pages is a better input than a crawl of your homepage. That reasoning is sound. It just describes a problem that the model vendors have chosen to solve differently.

The companion file, /llms-full.txt, concatenates the actual content of those pages into a single markdown document rather than linking out. Originality.ai measured llms-full.txt growing from 23 sites to 2,463 over the same period, a 107x increase off a tiny base.

What actually reads it

This is the part most guides skip. Ahrefs looked at server logs across 137,000 domains for May 2026 and found that 97% of llms.txt files were never requested at all. Among the 3% that saw any traffic, the requesters break down like this:

Requester	Share of llms.txt requests	Is it an AI engine?
SEO audit tools	21.7%	No
Unidentified bots	14.9%	Unknown
General web crawlers	13.1%	No
GPTBot	4.51%	Yes, training class
ClaudeBot	0.80%	Yes, training class
DeepseekBot	0.02%	Yes
AI retrieval bots, aggregate	1.1%	Yes, retrieval class

The largest single consumer of llms.txt is the category of tools that audit whether you have an llms.txt. That is close to a self-referential market. The retrieval crawlers, the ones that actually decide whether you get cited in a live answer, account for roughly one request in ninety.

The vendor statements line up with the logs. John Mueller, then at Google, wrote in June 2025 that no AI system was using llms.txt, comparing it to the keywords meta tag: a description of what a site owner claims their site is about, unverified against the site itself. On June 15, 2026, Google added a note to its AI optimization guidance stating that llms.txt files are not required for Google Search. OpenAI's crawler documentation directs site owners to robots.txt. Anthropic's does the same.

llms.txt against the files that do work

	robots.txt	sitemap.xml	llms.txt	llms-full.txt
Status	De facto standard since 1994	Standard, documented by Google and Bing	Proposal, September 2024	Proposal, companion
Honoured by major crawlers	Yes	Yes	No	No
What it controls	Access	Discovery	Nothing. It suggests	Nothing. It supplies
Format	Plain text directives	XML	Markdown	Markdown
Failure mode if wrong	Severe. You deindex yourself	Moderate. Slower discovery	None. It is ignored	None, plus wasted bandwidth
Effort to maintain	Low	Automated	Manual, drifts fast	High if hand-maintained

The row that matters is the last one. robots.txt and sitemap.xml are cheap to keep correct. An llms.txt is a hand-curated index that goes stale the moment your docs change, and a stale index is worse than none because the small number of systems that do fetch it get a confidently wrong picture of your product.

Why "the new robots.txt" is the wrong frame

The phrase shows up in most llms.txt guides and it inverts the actual relationship. robots.txt works because it is enforcement-adjacent: crawler operators honour it, and the ones that do not get blocked at the edge and lose access entirely. Its authority comes from thirty years of mutual dependence between crawlers and publishers.

llms.txt has no such mechanism. It is a publisher making a claim about their own content to a consumer with no obligation to read it and no way to verify it. Mueller's comparison to the keywords meta tag is precise, and the keywords meta tag is a useful precedent for exactly how this ends: a self-declared summary that search engines eventually ignored because self-declaration is trivially gamed.

If llms.txt does get adopted, it will most likely be because a model vendor decides curated markdown is cheaper to process than a full crawl, not because publishers adopted it hard enough to force the issue. Adoption pressure runs the wrong direction here.

The case for shipping one anyway

Three arguments survive the data.

Option value at near-zero cost. A good llms.txt takes about 30 minutes if your documentation is already organised. If any engine turns it on, you are already there. If none does, you spent half an hour. That is a reasonable bet, and it is a completely different claim from the one most guides make.

It forces an inventory. Writing the file means deciding which 20 pages actually represent your product. Most security vendors cannot answer that question quickly, and the exercise of answering it improves the pages themselves. The artifact is worth less than the work of producing it.

Agent traffic is a different question from search traffic. Coding agents and internal RAG pipelines built by your own customers do fetch predictable paths, and they are not bound by what OpenAI or Google chose to implement. If your product has an API and customers build against it, a clean markdown index has a plausible consumer that has nothing to do with AI search.

One honest complication, since it is my own. In a 90-day citation study I ran in early 2026 I recorded an 11% lift concentrated on Claude and Perplexity after publishing llms.txt across four properties. I no longer think that number supports a causal claim. Several changes shipped in the same window, the design did not isolate them, and the large-scale log evidence since points the other way. Publishing a result and then arguing against it is uncomfortable and it is the correct read of the data.

What does not survive: the claim that llms.txt improves your AI visibility. There is no evidence for it, the log data argues against it, and treating it as a visibility lever displaces work that does move citations. Structuring pages for extraction moves citations. Schema moves citations. Being reachable by the retrieval crawlers moves citations. An llms.txt does not.

A working llms.txt for a security vendor

Concrete beats abstract. This is what a good one looks like for a hypothetical CIAM vendor, and the structure transfers to any security product.

# Acme Identity

> Acme Identity is a customer identity and access management (CIAM) platform for
> B2B SaaS companies. It provides SSO, MFA, passwordless authentication, and
> SCIM provisioning, deployed as a hosted service or in a customer VPC.

Acme is priced per monthly active user with a free tier to 10,000 MAU.
It is SOC 2 Type II and ISO 27001 certified. FedRAMP status: in process.

## Product

- [Architecture overview](https://example.com/docs/architecture): tenancy model, data residency, failure domains
- [Authentication methods](https://example.com/docs/auth-methods): passwords, passkeys, magic links, TOTP, WebAuthn
- [SCIM provisioning](https://example.com/docs/scim): supported attributes and known IdP quirks
- [Pricing](https://example.com/pricing): per-MAU tiers, what counts as an active user

## Implementation

- [Quickstart](https://example.com/docs/quickstart): first login in 15 minutes
- [Migration from Auth0](https://example.com/docs/migrate-auth0): password hash import, downtime expectations
- [Self-hosted deployment](https://example.com/docs/self-hosted): requirements and limits

## Compliance

- [SOC 2 and ISO 27001](https://example.com/trust): scope, audit dates, report request
- [Data residency](https://example.com/docs/residency): available regions and what stays in-region
- [Subprocessors](https://example.com/trust/subprocessors): current list with dates

## Optional

- [Changelog](https://example.com/changelog)
- [Engineering blog](https://example.com/blog)

Four things make this file better than most:

1. The blockquote answers "what is this" in one sentence with the category name in it. Not a tagline. A definition.
2. Every link carries a note saying what is in the page. A bare list of URLs supplies no more information than a sitemap.
3. Facts a buyer actually asks about sit above the fold: pricing model, certifications, deployment options. These are the attributes that show up in comparison prompts.
4. FedRAMP status says "in process" rather than being omitted. Stating a negative or partial fact plainly is what makes the rest of the file credible, and it prevents a model from filling the gap with a guess.

Validating and monitoring

If you ship one, hold it to the same standard as any other public artifact.

1. Confirm it serves as text/plain or text/markdown, not text/html. A framework that wraps it in a page shell defeats the point. Curl it and look at the raw bytes.
2. Check every link returns 200. A curated index with dead links is worse than no index. Automate this in CI alongside your other link checks.
3. Regenerate it from source. Hand-maintained files drift within a quarter. Generate it from your docs tree so it cannot silently diverge.
4. Log requests to /llms.txt and review quarterly. This is the only way you will personally know if adoption changes, and given the current numbers, it is also a fast way to confirm that nothing is reading it.

Then go do the work that actually moves the number. Documentation structure outperforms every file at your site root, and crawl budget determines whether any of it gets read in the first place.

Frequently Asked Questions

Does llms.txt actually work in 2026?

Not for AI search visibility. Ahrefs' analysis of 137,000 domains found 97% of llms.txt files received zero requests in May 2026, and among the files that were fetched, AI retrieval bots accounted for roughly 1.1% of requests. Google has stated llms.txt is not required for Search. There is no published evidence linking an llms.txt file to increased citations.

Is llms.txt the new robots.txt?

No, and the comparison is misleading. robots.txt is honoured by every major crawler and controls access. llms.txt is an unenforced publisher claim that no major engine currently parses. John Mueller's comparison to the keywords meta tag is closer: a self-declared summary of what a site owner says their site is about, with no verification against the site itself.

Should I create an llms.txt file?

Yes if you have real documentation and 30 minutes, on option-value grounds rather than expected-return grounds. No if creating it displaces work on page structure, schema markup, or crawler access, all of which have measurable effects on citations today.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a curated index that links to your best pages with a note on each. llms-full.txt concatenates the actual content of those pages into a single markdown document, so a model can consume everything in one fetch. Originality.ai measured llms-full.txt on 2,463 sites in May 2026 against 36,120 for llms.txt.

How many websites have an llms.txt file?

Originality.ai tracked 36,120 llms.txt files across a monitored set of more than 3 million websites in May 2026, up from 4,088 in June 2025. That is an 8.8x increase in a year, and still roughly one site in a hundred within that sample.

Do OpenAI, Anthropic, or Google read llms.txt?

Not in any documented capacity. All three point site owners to robots.txt for crawler management. GPTBot and ClaudeBot appear in llms.txt request logs at 4.51% and 0.80% of requests to the small fraction of files that get fetched at all, which is consistent with incidental crawling rather than deliberate parsing.

What should I do instead of llms.txt?

Three things, in order. Confirm retrieval crawlers can reach you at all, since a WAF rule beats any file at your site root. Structure pages so individual sections survive being extracted out of context. Add schema markup that names your product, category, and organisation explicitly. Each of these has evidence behind it that llms.txt does not.