Measuring Whether AI Assistants Mention You

There is no rank to track. An assistant’s answer is sampled, personalised and session-dependent, so the only honest measurement is a proportion with an error bar — and the arithmetic below shows that the error bar at typical sample sizes is wider than every improvement this industry reports.

Why a rank tracker cannot exist here

A classical rank tracker works because a search result page for a given query, region and device is close to deterministic. Ask twice, get the same ten links. Four properties of assistant output break every part of that.

• Sampling. Generation draws from a distribution. Identical inputs produce different text, and different text can cite different sources — see why a model returns a distribution rather than an answer.
• Personalisation. Account memory, prior turns, location and product tier all change the answer, and none of them can be held constant across users.
• Retrieval variance. The query rewriter may issue different searches on different runs, and the backend index changes underneath.
• Version drift. Models and prompt templates are updated without announcement, so a series measured over months is not measuring one system.

Any product presenting a single “your AI visibility score is 62” has collapsed all of that into one number and thrown away the uncertainty. The number is not wrong so much as meaningless without the interval.

What you can measure instead

One well-defined quantity: the proportion of runs, under stated conditions, in which a stated prompt produces an answer that mentions you or cites your domain. That is a binomial proportion. It has an estimator, it has a confidence interval, and two measurements can be compared properly.

Everything else worth having is a secondary field recorded alongside it: which URL was cited, whether the mention was accurate, whether a competitor was mentioned in the same answer, and its position in the answer. Those are descriptive; the proportion is the thing with statistics attached.

The protocol

1. Fix a prompt set and never edit it. Twenty to fifty prompts covering the questions where you would expect to be a reasonable answer. Write them as a real user would, not with your brand name in them — a prompt containing your name measures nothing but whether the assistant can read.
2. Fix the conditions and write them down. Logged out or a clean account with memory disabled, stated country, stated model and version string, stated date, stated interface (API or product UI — they differ, and the API usually has no retrieval at all unless you enable it).
3. Fix the number of runs per prompt before you start, using the arithmetic in the next section. Decide it in advance so you cannot stop when the number looks good.
4. Define the outcome before you look. Exactly what counts as a mention: your brand name in the prose, your domain in a citation, or either. Write it down. Ambiguity here is where wishful counting enters.
5. Record the raw output, not just the verdict. Full text and full source list per run, timestamped. When a figure is challenged six months later, the raw data is the only thing that answers it.
6. Report the proportion with its interval, never the proportion alone.
7. Re-run the identical protocol to compare. A changed prompt set makes the comparison meaningless, and it is the most common way these studies quietly break.

How many samples you actually need

This is the section that matters, and it is ordinary arithmetic you can check.

A mention rate is a binomial proportion. For an observed proportion p over n independent runs, the standard error is the square root of p(1-p)/n, and a 95% interval is roughly p ± 1.96 × SE. Work a realistic case: thirty runs, twelve mentions.

n  = 30
k  = 12
p  = 12 / 30                     = 0.400

SE = sqrt(p(1-p)/n)
   = sqrt(0.400 × 0.600 / 30)
   = sqrt(0.24 / 30)
   = sqrt(0.008)                 = 0.0894

95% interval
   = 0.400 ± 1.96 × 0.0894
   = 0.400 ± 0.175
   = [0.225, 0.575]

The measured rate is 40% and the honest statement is somewhere between about 22% and about 58%. An agency reporting that it moved you from 30% to 40% has reported a difference that sits entirely inside that interval. It is not evidence of a change. It is what a coin does.

Now invert it. To measure a proportion near 0.5 to within five percentage points at 95% confidence, using the worst case p(1-p) = 0.25:

n = 1.96² × p(1-p) / margin²
  = 3.8416 × 0.25 / 0.05²
  = 0.9604 / 0.0025
  = 384 runs

...per condition. Comparing before and after needs both:
  384 × 2 = 768 runs

For a ±10 point margin:
  3.8416 × 0.25 / 0.01 = 96 runs per condition, 192 total.

So the practical thresholds are: about 100 runs per condition to see a ten-point change, about 400 to see a five-point change. With a fifty-prompt set that is two runs per prompt for the coarse version and eight for the fine one — achievable, and nothing like what a dashboard refreshing daily on one run per prompt is doing.

One more caution: runs of the same prompt are not fully independent if the backend caches or if session state leaks between them. Fresh session per run is not fastidiousness, it is what makes the arithmetic apply at all.

Running it

The harness is deliberately dull. It reads a prompt file, runs each prompt n times against a chat completions endpoint, records everything, and prints proportions with intervals.

#!/usr/bin/env node
// mention-rate.mjs — a mention rate with an error bar.
//   node mention-rate.mjs prompts.txt 8 > runs.jsonl
//
// One prompt per line in prompts.txt. Set BASE_URL, API_KEY and MODEL
// for whichever endpoint you are sampling. Record the model string and
// the date in your write-up; they are part of the measurement.

import { readFileSync, appendFileSync } from "node:fs";

const BASE_URL = process.env.BASE_URL;        // e.g. an OpenAI-compatible base
const API_KEY  = process.env.API_KEY;
const MODEL    = process.env.MODEL;

const BRAND  = /\bexample(\.com)?\b/i;      // your brand, as a regex
const DOMAIN = /example\.com/i;              // your domain in a citation

const [file, runsArg] = process.argv.slice(2);
const RUNS = Number(runsArg ?? 8);
const prompts = readFileSync(file, "utf8").split("\n")
  .map((s) => s.trim()).filter(Boolean);

async function ask(prompt) {
  const res = await fetch(BASE_URL + "/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: "Bearer " + API_KEY,
    },
    body: JSON.stringify({
      model: MODEL,
      messages: [{ role: "user", content: prompt }],
    }),
  });
  if (!res.ok) throw new Error(res.status + " " + (await res.text()).slice(0, 200));
  const json = await res.json();
  return json.choices[0].message.content ?? "";
}

const results = new Map();   // prompt -> { n, k }

for (const prompt of prompts) {
  results.set(prompt, { n: 0, k: 0 });
  for (let i = 0; i < RUNS; i++) {
    let text = "";
    try { text = await ask(prompt); }
    catch (e) { console.error("skip:", e.message); continue; }

    const hit = BRAND.test(text) || DOMAIN.test(text);
    const r = results.get(prompt);
    r.n++; if (hit) r.k++;

    // Raw output, always. This is the part that makes it auditable.
    appendFileSync("runs.jsonl", JSON.stringify({
      at: new Date().toISOString(), model: MODEL, prompt, run: i, hit, text,
    }) + "\n");
  }
}

function interval(k, n) {
  if (n === 0) return [0, 0, 0];
  const p = k / n;
  const se = Math.sqrt((p * (1 - p)) / n);
  return [p, Math.max(0, p - 1.96 * se), Math.min(1, p + 1.96 * se)];
}

let K = 0, N = 0;
for (const [prompt, r] of results) {
  const [p, lo, hi] = interval(r.k, r.n);
  K += r.k; N += r.n;
  console.log(
    (p * 100).toFixed(0).padStart(3) + "%",
    "[" + (lo * 100).toFixed(0) + "–" + (hi * 100).toFixed(0) + "]",
    r.k + "/" + r.n,
    prompt.slice(0, 60),
  );
}
const [p, lo, hi] = interval(K, N);
console.log("\nOVERALL " + (p * 100).toFixed(1) + "%  95% CI ["
  + (lo * 100).toFixed(1) + ", " + (hi * 100).toFixed(1) + "]  n=" + N);

Two warnings about what this measures. An API endpoint usually has no web retrieval unless you turn it on, so by default you are measuring what the model knows from training rather than what a user of the consumer product sees — those are different questions and both are worth asking, but you must say which one you asked. And running a product UI by automation may breach its terms of service; check before you build that, and prefer the documented API.

The passive half: referrers and fetches

Sampling tells you what an assistant says. Your logs tell you what actually happened, at no cost and with no sampling error, and the two answer different questions.

Set the log side up first. It is cheaper, it accumulates history while you argue about protocol, and the crawler audit already produces most of it.

Reading a result honestly

Four rules that will keep you from publishing something you have to retract.

• Never report a proportion without its n and its interval. “40%” is not a result; “40%, 95% CI 22–58, n=30” is.
• Do not compare across model versions. If the provider updated the model between your two measurements, the difference is confounded with the update and no analysis separates them.
• Do not attribute a change to your intervention. You have no control group. The strongest honest sentence is that the rate changed over a period during which you also did X.
• Report the failures. Prompts where the assistant named a competitor, or gave a wrong fact about you, are more actionable than the aggregate — and the second kind leads directly to correcting a false claim.