We Tracked 180,000 AI Answers. Here Is What We Found

Most teams treat LLM output as deterministic. Run the same prompt twice, get roughly the same answer, ship it.

That assumption is doing an enormous amount of load-bearing work, and it is false in at least four separate ways.

I know because we measured it. For 18 months we have been running production citation tracking across ChatGPT, Perplexity, Google AI Overviews, and Microsoft Copilot. Roughly 180,000 responses per customer per quarter. What we found is messier than the AI engineering community discusses publicly, and it has direct consequences for anyone whose buyers now start their research inside a language model.

This post is the long-form version of a talk I gave on the CloudX Expo Innovation Stage at AI TechWorld 2026 on 3 September. It includes the architecture we built first and had to throw away.

The short answer

Four findings, up front, before the argument that supports them:

• AI visibility can fall more than 30% inside six hours with no model update, because the retrieval index moved.
• ChatGPT and Perplexity cite the same domains only about 11% of the time for identical queries in the same window.
• Roughly 17% of citations point at the brand's own domain. The other four in five point at a corpus the brand cannot edit.
• Citation change is four different events on four different clocks, and treating them as one is why alerting fails.

How this was measured. Persona-mapped prompt sets, run hourly against ChatGPT, Perplexity, Google AI Overviews, and Microsoft Copilot, with the engine, model, prompt, persona, and timestamp recorded on every stored response. Roughly 180,000 responses per customer per quarter, held append-only over 18 months, with drift measured as a windowed comparison against a variance baseline computed separately for each engine. The numbers below are from that dataset, not from a vendor survey or a one-off benchmark run.

The question that turns out to be hard

Every enterprise customer eventually asks the same thing: am I being cited correctly?

It sounds like a search question. It is not. It is an observability question, and answering it honestly requires you to add three qualifiers the customer did not ask for: cited correctly by whom, on which engine, and as of when?

Take away any one of those three and the answer becomes meaningless. Here is why.

Finding 1: Visibility can drop 30% overnight with no model change

In one tracked window, Copilot's citation set for a group of cybersecurity queries shifted more than 30% within six hours. A news event had occurred. Competing content got picked up. Brands that had been cited in nearly every relevant answer dropped out of the majority of them.

There was no model update. No version bump, no changelog, no announcement. The underlying retrieval index moved and the model dutifully reported what it now found.

If you were polling daily, you recorded one number that morning and one number the next morning. Depending on where those samples landed, you either saw nothing at all or you saw a change with no explanation attached to it. The event itself, the thing you actually needed to know about, happened entirely between your measurements.

This is the first structural problem with how most teams approach AI visibility. Cadence is not a tuning parameter you optimize for cost. Cadence determines the class of events you are capable of observing at all.

Finding 2: Winning one engine tells you almost nothing about the other three

Across our tracked B2B queries, ChatGPT and Perplexity agree on approximately 11% of cited domains. Same questions. Same time window. Almost entirely different source sets.

Sit with that number for a moment, because it invalidates a lot of common practice.

It means "we rank first in ChatGPT" is not a statement about your AI visibility. It is a statement about one engine, and it carries almost no predictive information about the other three. You could be the single most-cited vendor in one engine and completely absent from another, and a single-engine dashboard would show you nothing but green.

It also means that averaging across engines is worse than useless. Blend four engines into one score and a 40-point gap on one of them disappears into a healthy-looking number. The average is arithmetic, not information. If you are choosing a monitoring tool, this is the first question to ask it: does it report per engine, or does it hand you a blended score? I went through what the current tools actually measure in The Best AI Visibility Tools in 2026.

The reason this matters commercially is simple: your buyer only ever sees one engine, and you do not get to pick which one. If you are absent from the engine they happen to use, you are absent for that deal. There is no second place in a single synthesized answer.

Finding 3: Only 17% of your citations come from your own domain

This is the finding that surprises people most.

Across 180,000 tracked responses, the distribution of cited domains broke down roughly as follows: review and comparison sites accounted for the largest share, followed by community forums and archives, then editorial and trade press. Owned domains, meaning your documentation, your blog, and your product pages, accounted for about 17%. Developer docs and public repositories made up the remainder.

Four in five citations point at a corpus you do not own and cannot edit.

For anyone who came up through traditional search optimization, this is a significant reframing. The content you control is a minority stake in how you are represented. Most of what an engine says about you is assembled from what other people wrote, in places you can influence but not author. It is also why citation share is a more honest board metric than rank: it counts the whole corpus, not the slice you publish.

Finding 4: Most drift is noise, and some of it is your pipeline leaving

Once you are measuring continuously, you discover that "the citation set changed" is not one event. It is at least four, and they behave completely differently.

Sampling noise operates on the scale of minutes. Temperature, routing, and load balancing produce small variations with no underlying cause. This is the natural noise floor of the system. If you alert on it, you will train your team to ignore alerts within two weeks.

Index drift operates on the scale of hours. The retrieval corpus refreshes, new content gets indexed, older content ages out. This is real signal. It is also where most competitive displacement actually happens.

Event-driven shifts operate over six to twenty-four hours. A breach disclosure, a funding announcement, a CVE, a product launch. The corpus reorganizes around the news and citation sets follow. This class is unique because it comes with a window: while the corpus is actively being rewritten, well-timed content has a disproportionate chance of being picked up.

Model updates operate over weeks and are frequently silent. A version changes underneath you and every historical score you have collected becomes non-comparable. Baselines must be re-established from scratch.

Drift class	Timescale	Cause	Correct response
Sampling noise	Minutes	Temperature, routing, load balancing	None. Fold it into the baseline and never alert.
Index drift	Hours	Retrieval corpus refreshes, content ages in and out	Same-day content response. Diff the set, ship a counter-asset.
Event-driven shift	6 to 24 hours	Breach, funding round, CVE, launch	Act within hours, while the corpus is still being rewritten.
Silent model update	Weeks	Provider changes the model with no changelog	Re-baseline every prompt. Historical comparisons are void.

The expensive mistake, the one we made, is treating all four as the same event and alerting on any change. What you actually need is a variance baseline per engine, because each engine has a different natural noise floor, and a classification step that routes each detected change to the right response.

Models and versions are part of the measurement, not a detail

One thing I would add to the talk, because it came up in almost every hallway conversation afterwards: the model you query is part of the measurement, and most tracking setups do not record it.

"We track ChatGPT" is not a specification. ChatGPT is a product surface that routes to different models, with different retrieval behaviour, different citation formatting, and different defaults for logged-in and logged-out users. The same is true of Copilot, of AI Overviews across query classes, and of Perplexity across its model selector. A number collected under one configuration is not comparable to a number collected under another, and if you did not write the configuration into the row, you cannot tell afterwards which one you had.

Three rules fall out of that:

• Record the engine, the model, and the configuration on every row. Engine alone is not enough. When a provider swaps the default model underneath a product surface, the only way to see it is a break in your own series that lines up with nothing you did.
• Query the same prompt multiple times a day per engine. One sample per day is one draw from a distribution, reported as if it were the distribution. Ten draws across the day gives you a mean and a variance, and the variance is what tells you whether tomorrow's move is real.
• Segment before you aggregate. Per engine, per model, per persona, per query intent. Aggregate at the top for the executive view, but keep the segments underneath, because the action always lives in a segment and never in the average.

This is the difference between a number that decorates a dashboard and a number somebody can act on. If the report says visibility fell, the next three questions are which engine, which model, and which set of prompts. A tracker that cannot answer those has told you that something happened, which is roughly as useful as a smoke alarm with no location.

The architecture we built first, and why it failed

Our first system was the obvious one. Poll each engine daily. Diff the citation set against yesterday. Alert when it changes.

It failed in three ways, and I am describing them in detail because it is the version most teams build first.

Alert fatigue arrived in week two. Daily polling caught natural sampling variance and reported it as drift. Because we had no per-engine variance baseline, we had no way to distinguish a meaningful shift from the noise floor. Every alert looked equally urgent, so within days none of them were.

We had no causal signal. We could see that a citation set changed. We could not tell whether it was a model update, a news event, a competitor's new content, or nothing at all. A change notification with no attached cause does not produce action; it produces a Slack thread that goes nowhere.

Scraping was the wrong primitive. Each engine presents citations differently. Different surfaces, different formats, different failure modes. We were maintaining four fragile scrapers rather than operating one system, and the maintenance burden grew faster than the value.

What we run now

The current architecture has four stages, and only two of the changes actually mattered.

Query distribution. Persona-mapped prompt sets running hourly, with jittered scheduling to avoid engine-side caching artifacts. The prompts are mapped to buyer roles, because a security director in the education stage of a purchase asks structurally different questions than the same person in the evaluation stage. Both matter, and they surface different competitive sets. One buyer question also rarely stays one query inside the engine, which is the subject of Query Fan-Out.

Citation extraction. A normalized schema across all four engines. For every response we capture the engine, the model and configuration, prompt identifier, persona, timestamp, raw response, cited domains, positions, and whether the tracked brand appeared at all.

Storage. Append-only. Every response is a row and nothing is ever overwritten. This sounds wasteful until you realize the obvious alternative destroys your ability to do the one thing the system exists for. You cannot detect drift in data you have already replaced.

Drift detection. Windowed comparison against a variance baseline computed per engine, with alerting on statistical anomaly rather than on any change at all.

The two changes that mattered were hourly cadence and per-engine variance baselines. Everything else is plumbing.

What measuring this actually costs

The scale surprises people, so here is the arithmetic for a single mid-sized customer at a conservative cadence.

Fifty tracked prompts, four engines, ten queries per prompt per day. That is 2,000 responses per day, 60,000 per month, and 180,000 per quarter. Per customer.

Every one of those is a row you keep, containing the raw response alongside the extracted citation data. The storage cost is trivial by modern standards. The analysis layer on top of it is the entire product.

Detection without a response plan is theater

Knowing that a number moved is not useful on its own. Each drift class needs a defined response, and the useful insight is that they are not all incidents.

• Sampling noise requires no response. Fold it into the baseline and never alert.
• Index drift requires a same-day response from whoever owns content. Diff the citation set, identify what replaced you, and ship a counter-asset.
• Event-driven shifts require action within hours, because the window closes. This is the one class that is genuinely an opportunity rather than a problem.
• Model updates require re-baselining every tracked prompt, and an acknowledgment that historical comparisons are void.

Two of these four are opportunities. That reframe changed how we staffed the function.

What this means if you are building on LLM output

Four assumptions we had to delete from our own thinking, all of which generalize past citation tracking to any system built on model output.

"The model returned X" is incomplete. Every number you show a user needs a timestamp, an engine, and a model attached. A bare value is a claim you cannot defend when someone asks you about it three weeks later.

"Our eval suite passes" is incomplete. Passed against which baseline version? A silent model update makes every historical score non-comparable. Version your baselines like you version code.

Caching without a defensible TTL is a liability. An answer cached past its drift window is worse than having no answer, because it is confidently wrong.

Correctness is a distribution, not a value. Build assertions that accept variance within a known range and flag statistical anomaly, rather than assertions that demand exact reproduction.

Where to start

If any of this is relevant to your product, the entry cost is lower than it looks.

Pick twenty prompts your buyers genuinely ask. Query four engines. Record the engine and model on every row. Store every response without overwriting anything. Run it for a week.

You will see drift that nobody told you about. That is the whole first milestone, and it is enough to justify whatever you build next.

The larger point is this: LLM output is not deterministic and it is not stable over time. If citation accuracy matters to your product or your pipeline, it needs observability infrastructure, not a benchmark you run once and file away.

Frequently Asked Questions

Do LLMs give the same answer to the same question every time?

No. Across 18 months of production tracking on ChatGPT, Perplexity, Google AI Overviews, and Microsoft Copilot, the same prompt returns different answers across engines, across time, and across silent model updates. Variation is the normal state, not an error condition.

How much can AI visibility change in a single day?

We measured a citation set shift of more than 30% within a six hour window on one engine following a news event, with no model update involved. The underlying retrieval index changed.

Do different AI engines cite the same sources?

Rarely. ChatGPT and Perplexity overlap on approximately 11% of cited domains for identical queries in the same time window. Measuring one engine tells you very little about the others.

What percentage of AI citations come from a brand's own website?

Approximately 17% in our dataset. The majority come from review sites, community forums, and editorial coverage, meaning most of what an AI engine says about a company is assembled from sources that company does not control.

How often should you monitor AI citations?

Hourly, or at minimum several times a day per engine. Daily polling structurally cannot observe sub-day events, which are both real and frequent. A daily sample can record stability while a 30% shift occurs and partially recovers between measurements.

Does the specific model matter, or is tracking the engine enough?

The model matters. A product surface like ChatGPT or Copilot routes to different models with different retrieval behaviour and different citation formatting, and providers change the default without announcing it. Record the engine, the model, and the configuration on every stored response, or you will not be able to tell a provider change from your own.

What is the difference between AI citation noise and real drift?

Sampling noise operates over minutes and has no underlying cause. Index drift operates over hours and reflects genuine corpus change. Distinguishing them requires a variance baseline computed separately for each engine, because each has a different natural noise floor.

This post accompanies a talk given on the CloudX Expo Innovation Stage at AI TechWorld 2026 on 3 September. The session page has the program, the takeaways, and the reference architecture in short form. GrackerAI, my company, builds the citation observability platform described here.