Implementing llms.txt and AI Crawler Signals

AI crawlers are not one thing. Some collect pages to train models. Some build the search index an AI assistant cites from. Others fetch a single page because a user asked about it right now. Each major AI company now runs separate user agents for these jobs, and robots.txt can treat them differently. Getting this split right is a core GEO skill.

This chapter covers robots.txt directives for AI crawlers, the llms.txt proposal, and the structured signals that help AI engines understand your content.

Understanding AI Crawlers

Before configuring anything, you need to know which crawlers exist and what they do. The table uses each operator's own documentation, checked in November 2026.

User agent	Operator	What it does	Affects citations?
OAI-SearchBot	OpenAI	Builds the index for ChatGPT search; opted-out sites are not shown in ChatGPT search answers	Yes
ChatGPT-User	OpenAI	Fetches pages for user-initiated actions; not automatic crawling; robots.txt may not apply	Indirectly
GPTBot	OpenAI	Collects content that may be used to train OpenAI's foundation models	No (training only)
Claude-SearchBot	Anthropic	Crawls to improve Claude's search results (Anthropic help center)	Yes
Claude-User	Anthropic	Fetches pages when a Claude user asks a question	Indirectly
ClaudeBot	Anthropic	Collects content that may be used for model training	No (training only)
PerplexityBot	Perplexity	Indexes sites for Perplexity search results; not used for foundation model training (Perplexity docs)	Yes
Perplexity-User	Perplexity	Fetches pages for a user's question; generally ignores robots.txt	Indirectly
Googlebot	Google	Crawls for Google Search, including AI Overviews and AI Mode	Yes
Google-Extended	Google	A robots.txt token, not a separate crawler; controls use in Gemini training and grounding. Does not affect Search inclusion or ranking	Not for Google Search
Bingbot	Microsoft	Crawls for Bing, which grounds Microsoft Copilot answers	Yes
Applebot / Applebot-Extended	Apple	Applebot powers Siri and Spotlight search; Applebot-Extended only controls training of Apple's models (Apple)	Applebot yes; Extended no
Meta-ExternalAgent	Meta	Crawls for uses such as training foundation AI models (Meta)	Mostly training

Two common mistakes come from this table. First, blocking GPTBot does not remove you from ChatGPT search, and allowing it does not put you there; OAI-SearchBot does that job. Second, Google-Extended has no effect on AI Overviews or AI Mode, which are part of Google Search and governed by Googlebot. To limit what Google shows from a page in Search, including AI features, Google points to nosnippet, data-nosnippet, max-snippet, or noindex.

The Crawler Permission Decision

Treat search crawlers and training crawlers as two separate decisions.

Search and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot) decide whether you can be cited at all. For a B2B SaaS company that wants AI visibility, allow them on your public content.

Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent) decide whether your pages may be used to train future models. This is a business and legal judgment. My view for most B2B marketing content is to allow them, because being in training data helps models know your brand when they answer without searching. Companies with proprietary research or licensing concerns may reasonably block them.

User-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) act for a person who asked about your page. Their operators say robots.txt may not apply to them, so do not rely on robots.txt to stop them.

Configuring robots.txt for AI Crawlers

Your robots.txt file sits at the root of your domain. Write a group for each AI crawler you want to treat differently from the default.

Example robots.txt Configuration

# Default for all crawlers, including search engines
User-agent: *
Disallow: /account/
Disallow: /admin/
Disallow: /api/
Disallow: /internal/

# AI search and retrieval crawlers: allow public content
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /account/
Disallow: /admin/
Disallow: /api/
Disallow: /internal/

# AI training crawlers: allow or block as a business decision
# (this example allows them on public content)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /account/
Disallow: /admin/
Disallow: /api/
Disallow: /internal/

# Example of blocking a training crawler entirely
User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Key Principles

1. Remember how groups work: A crawler follows only the most specific group that names it, and ignores the * group. So every named group must repeat the Disallow lines it needs.
2. Allow your best content: Do not accidentally block /blog/, /guides/, or /docs/ for the search crawlers you want citations from.
3. Block sensitive paths: Keep account pages, admin areas, API endpoints, and internal tools disallowed. robots.txt is a request, not security, so protect these with authentication too.
4. Update regularly: AI companies add and rename user agents (OpenAI, Anthropic, and Perplexity have each added separate search and user-request agents alongside their training crawlers). Review your robots.txt quarterly against each operator's documentation.

The llms.txt Specification

llms.txt is a proposal published by Jeremy Howard on September 3, 2024. It gives language models a concise markdown map of your most important content. Think of it as a curated reading list for AI tools, not a sitemap replacement and not an access-control file.

Be realistic about adoption. As of November 2026, I have not found a major AI search engine documenting that it reads llms.txt when deciding what to cite. Developer tools and AI agents that fetch documentation use it more. It is cheap to publish, so publish it, but do not expect it to move citations on its own.

File Location and Format

Place the file at https://yourdomain.com/llms.txt. The spec defines a simple markdown structure, in this order:

1. One H1 with the name of your site or project. This is the only required element.
2. A blockquote with a short summary.
3. Optional paragraphs or lists with more context (no headings).
4. Zero or more H2 sections, each a list of links in [name](url): notes form.
5. An H2 named Optional, by convention, for secondary links an agent can skip.

The spec also suggests serving a clean markdown copy of key pages at the same URL with .md appended, and linking it from the page with rel="alternate" type="text/markdown".

Complete llms.txt Template

# [Your Company Name]

> [One or two sentences on what your company does and what these pages cover.]

[Optional paragraph explaining how the content is maintained or how to read it.]

## [Topic Cluster 1 Name]

- [Page title](https://yourdomain.com/page-1.md): [what this page covers and who it is for]
- [Page title](https://yourdomain.com/page-2.md): [description]

## [Topic Cluster 2 Name]

- [Page title](https://yourdomain.com/page-3.md): [description]

## Research and Data

- [Report title](https://yourdomain.com/research/report.md): [scope, sample size, methodology note]

## Optional

- [About us](https://yourdomain.com/about.md): [company overview]
- [Authors](https://yourdomain.com/authors.md): [who writes the content and their expertise]

Worked llms.txt Example

Acme Identity is a fictional company. Every number in its descriptions should be one you can back up on the linked page.

# Acme Identity

> Acme Identity builds customer identity and access management (CIAM) software for B2B SaaS companies. These pages are our reference material on enterprise authentication and customer identity.

Guides are maintained by our identity engineering team and reviewed when the underlying standards change.

## Enterprise Authentication

- [SAML vs OAuth 2.0 vs OIDC](https://acmeidentity.com/guides/saml-vs-oauth-vs-oidc.md): protocol comparison with implementation guidance
- [Passkeys for enterprise SSO](https://acmeidentity.com/guides/passkeys-enterprise.md): rolling out FIDO2 passkeys, with code examples
- [MFA deployment guide](https://acmeidentity.com/guides/mfa-best-practices.md): MFA deployment mapped to NIST SP 800-63B

## Customer Identity (CIAM)

- [CIAM guide for SaaS](https://acmeidentity.com/guides/ciam-complete-guide.md): architecture, vendor selection, and implementation
- [CIAM platform comparison](https://acmeidentity.com/comparisons/ciam-platforms-2026.md): feature-by-feature comparison with evaluation criteria

## Research and Data

- [Authentication benchmark 2026](https://acmeidentity.com/research/authentication-benchmark-2026.md): original survey data, with sample size and method stated on the page

## Optional

- [About Acme Identity](https://acmeidentity.com/about.md): company overview and compliance certifications
- [Auth0 vs Acme](https://acmeidentity.com/comparisons/auth0-vs-acme.md): head-to-head comparison

llms.txt Best Practices

Practice	Why It Matters
Keep it curated (I aim for 20 to 30 links)	The file is meant to be a concise selection, not a full index.
Update monthly	Remove dead links and add new key pages; a stale file points agents at outdated content.
Include descriptions	The spec allows notes after each link; they tell a model what the page is for before it fetches it.
Organize by topic	Topic clusters reinforce your domain authority in specific areas.
Note data sources	For research content, mention sample sizes and methodology. This builds trust.
Use absolute URLs	Always use full URLs, not relative paths. AI systems may process this file outside of browser context.

The llms-full.txt Companion File

Some sites also publish an llms-full.txt file with the full text of their key pages in one markdown document. It is a community convention, popularized by documentation platforms, rather than part of the llmstxt.org spec. It is most useful for coding assistants and agents that want a whole documentation set in one fetch.

llms-full.txt Structure

# [Your Company Name]: Full Content Index

## [Page Title 1]
URL: [Full URL]
Last Updated: [Date]
Author: [Name, Title]

[Full text content of the page, stripped of HTML/navigation/ads]

---

## [Page Title 2]
URL: [Full URL]
Last Updated: [Date]
Author: [Name, Title]

[Full text content of the page]

Structured Data Beyond Schema.org

While Schema.org is the primary structured data vocabulary, additional signals help AI engines understand your content.

Open Graph and Meta Tags for AI

Ensure every content page includes these meta tags:

<!-- Standard meta -->
<meta name="description" content="[150-160 char description with key entities]">
<meta name="author" content="[Author full name]">

<!-- Open Graph -->
<meta property="og:title" content="[Page title]">
<meta property="og:description" content="[Description matching meta description]">
<meta property="og:type" content="article">
<meta property="og:url" content="[Canonical URL]">
<meta property="og:site_name" content="[Brand name]">
<meta property="article:author" content="[Author profile URL]">
<meta property="article:published_time" content="[ISO 8601 datetime]">
<meta property="article:modified_time" content="[ISO 8601 datetime]">
<meta property="article:section" content="[Content category]">
<meta property="article:tag" content="[Topic tag 1]">
<meta property="article:tag" content="[Topic tag 2]">

Canonical URLs

Every piece of content should resolve to a single canonical URL. Duplicate content across multiple URLs splits attribution between copies.

<link rel="canonical" href="https://yourdomain.com/guides/saml-vs-oauth">

Ensure that:

• Every page has a canonical tag
• The canonical URL matches the URL in your llms.txt
• Redirects resolve to the canonical URL (no redirect chains)
• HTTP and HTTPS versions both resolve to the same canonical

Implementation Checklist

Use this checklist to verify your AI crawler signals are properly configured.

Signal	Status Check	Tool to Verify
robots.txt AI directives	Search crawlers allowed; training crawlers set by decision	Fetch yourdomain.com/robots.txt directly
llms.txt published	File accessible at domain root	Fetch yourdomain.com/llms.txt
llms.txt content current	All listed URLs are live and recently updated	Manual review or link checker
Schema.org on all content pages	Article, Person, Organization schema present	Schema Markup Validator (validator.schema.org)
FAQPage schema on FAQ sections	Structured data wraps visible FAQ content	Schema Markup Validator
Canonical URLs set	Every content page has a canonical tag	Screaming Frog or site crawl
Open Graph tags complete	og:title, og:description, og:type on all pages	Facebook Sharing Debugger or manual review
Meta dates accurate	published_time and modified_time match actual dates	Manual review of page source