Methodology
One AI answer isn’t a measurement
AI assistants don’t reliably give the same answer twice, even to the same question on the same day. So we measure with repetition, not screenshots.
What the research shows
Ask the same engine the same question twice on the same day, and the sources it cites overlap only partially (Jaccard 0.32–0.43), about as much as two different days do.
Checking whether a brand appears at all takes a handful of runs. A stable read on how often it's cited takes far more: from several dozen to over a hundred observations, depending on the platform.
What that means in practice
- Repetition
- We ask the same question more than once, so one lucky or unlucky answer doesn't pass for the truth.
- Controls
- We keep the conditions around each run the same, so what changes is the answer, not our setup.
- Ranges
- We report a range and how sure we are of it, not one number that claims more precision than the data has.
What we publish, and what we don’t
Findings are meant to be shared: observations, drift, and corrections to our own earlier calls.
What stays private is how a client’s engagement is built: which questions come first, how the work is sequenced, and the tooling behind it. The findings are the credibility. The methods are the work.
When we’re not sure, we say so
Sometimes repeated runs don’t settle. When an answer keeps moving with nothing else changed, we report that, because it tells you something: the category hasn’t settled yet.