How AI Search Actually Works (and Why Ranking #1 No Longer Matters)
GEO/AEO · intro · 8 min read · last reviewed 2026-08-20
AI engines assemble answers from three layers, not one ranking. Here is the parametric memory, retrieval index, live fetch model, an engine-by-engine comparison, and why blocking the wrong crawler token costs you a whole surface.
TL;DR
- AI answers come from three layers: parametric memory (6 to 18 months to change), a retrieval index (days to weeks), and live page fetch (instant). Diagnose which layer failed before you fix anything.
- An LLM compresses the internet's collective opinion about you, not just your own site; Muck Rack's 2026 analysis found roughly 84% of AI citations come from earned media, not brand-owned pages.
- Each engine favors different sources: ChatGPT leans on Wikipedia and high-authority sites, Google AI Overviews and Perplexity lean on Reddit and G2, Claude leans on documentation.
- Every AI lab ships multiple, independently governed crawler bots. Blocking one token does not block the others, and getting this wrong silently removes you from a fast-growing surface.
AI search engines do not rank pages, they assemble answers. An answer comes from three separate layers: what the model memorized during training, what it can retrieve from a live index right now, and what it fetches from a specific URL during the conversation. Most GEO programs stall because they diagnose the wrong layer.
The three layers, and why the split matters
Parametric memory is what the model absorbed while training. It has no web access at this moment; it is the model's gut instinct about your brand. It updates only when the vendor trains a new model, a lag of 6 to 18 months. You influence it by changing what the open web said about you before the next training cut.
Retrieval index is what the engine can look up right now. ChatGPT runs its own discovery index plus third-party search APIs. Google grounds Gemini and AI Mode against Google Search. Perplexity runs a proprietary index supplemented by Bing. Claude uses its own search index. This layer updates in days to weeks, and it is where most of your fast wins live.
Live fetch is the engine opening a specific URL during the conversation, usually because the user named it or the reranker wants confirmation. This updates instantly.
Test the layers separately or you will misdiagnose the problem. Ask an engine about your brand with web search off, then again with search on. If it praises a competitor with search off and cites you with search on, you have a training-data problem, not a content problem, and the fix runs on a different timeline. How to audit your AI visibility walks through the exact protocol.
The consensus principle
An LLM does not evaluate your website. It compresses the internet's collective opinion about you into three or four sentences. Your homepage is one vote among thousands. Reddit threads, review sites, analyst notes, listicles, YouTube transcripts, and Wikipedia are the other votes, and collectively they outweigh you. Muck Rack's May 2026 analysis of over 25 million links across ChatGPT, Claude, and Gemini citations found roughly 84% of AI citations came from earned media rather than brand-owned pages. This is why publishing more of your own content rarely fixes a visibility problem by itself. You do not win by writing better about yourself. You win by changing what everyone else writes about you.
What each engine actually favors
You cannot run one audit and call it done. A finding on ChatGPT does not transfer to Perplexity. Build a per-engine model of what the machine reads and what it rewards.
| Engine | Index | Update speed | Favors | Fastest lever |
|---|---|---|---|---|
| ChatGPT | Own discovery index + third-party search APIs + reading cache | Days | Wikipedia (7.8% of citations), credentialed media, sites with deep referring-domain profiles | Allow OAI-SearchBot, fix entity records, earn tier-1 mentions |
| Google AI Overviews | Google Search, query fan-out | Hours to days | Reddit, G2, schema-marked pages | Win fan-out sub-queries with passage-level answers |
| Google AI Mode | Google Search, deeper fan-out | Hours to days | Topical depth across a full content cluster | Publish clusters, not single pages |
| Perplexity | Proprietary index + Bing | Real time | Reddit, LinkedIn, G2, .gov, attributed data | Named-author original data, community presence |
| Claude | Own search index + live fetch | Days | Documentation, technical depth, primary sources | Allow all three Claude bots, publish real docs |
| Gemini app | Google Search, same fan-out as AI Mode | Hours to days | Same as AI Mode | Same as AI Mode |
| Copilot | Bing index, enterprise 365 distribution | Days | Bing-indexed, enterprise sources | Bing Webmaster Tools |
| Grok | X plus web | Real time | Practitioner discussion on X | Founder and team presence on X |
ChatGPT is the most risk-averse of the group. It cites what it can confidently attribute, so named authors, dates, and clean entity records matter more here than anywhere else. Perplexity sits at the other end, with the highest citation density of any engine and a real penalty for unattributed claims like "industry research shows." Claude is comparatively conservative about recommending a vendor outright and will often qualify a recommendation rather than assert it. Documentation quality and primary-source citation are the load-bearing signal there.
SE Ranking's study of 129,000 domains found sites with 32,000-plus referring domains were 3.5 times more likely to be cited by ChatGPT than sites under 200. It is the closest thing to a single authority signal that transfers across engines.
Crawlers are not one bot per company
Every major lab ships multiple, independently governed bots, and blocking one does not block the others. OpenAI alone runs GPTBot (training, blocking it has no effect on citations), OAI-SearchBot (the search index) and ChatGPT-User (live fetch during a conversation). OpenAI's own developer documentation states that a site disallowing OAI-SearchBot will not appear in ChatGPT search answers. Anthropic ships ClaudeBot (training), Claude-SearchBot (search index), and Claude-User (user-requested fetch) as three separate directives. Configuring robots.txt for every AI crawler is the full bot-by-bot build.
The agentic layer is coming
AI browsers and agents, Comet, Atlas, and Chrome's agentic browsing mode, do not read your page the way a human does. They act on it: filling forms, comparing prices, completing checkouts. This surface is small today. Structured, machine-readable product and pricing data is the early lever, and it will not stay optional.
Key takeaways
- Test with web search on and off separately. That split alone tells you whether you have a training problem or a retrieval problem.
- You do not win AI visibility by writing more about yourself. You win by changing what independent sources say about you.
- OpenAI's own documentation states that blocking OAI-SearchBot removes you from ChatGPT search answers entirely, a different bot from the GPTBot training crawler most teams think about.
- Sites with deep referring-domain profiles are measurably more likely to get cited by ChatGPT; authority signals carry across engines even though favored source types do not.
Frequently asked questions
- Why did my page rank number one and still get no AI citations?
- AI search retrieves passages for hidden sub-queries generated from the original question, not the page that ranks for the head keyword. A page can rank first and answer none of the actual retrieval targets.
- What is the difference between parametric memory and retrieval?
- Parametric memory is what the model learned during training and has no live web access; it changes only when a new model ships, typically 6 to 18 months. Retrieval is what the engine looks up right now, updating in days to weeks.
- Does blocking GPTBot hurt my ChatGPT citations?
- No. GPTBot collects training data only. The bot that controls ChatGPT search citations is OAI-SearchBot, a separate crawler with its own robots.txt directive.
- Which AI engine is easiest to get cited on?
- Perplexity has the highest citation density of any engine and rewards attributed, dated claims heavily, which makes it the fastest engine to move if your content already names its sources.
Related
Vendor comparisons
Go deeper