How to Track Your Brand Mentions in ChatGPT

Why one-off manual checks mislead

Large language model inference is not reproducible by default. Engineers at Thinking Machines Lab documented that even at temperature zero, batching and kernel-level non-determinism make identical requests return different completions — their write-up on defeating nondeterminism in LLM inference explains why the same prompt drifts run to run. The OpenAI API exposes a best-effort `seed` parameter and temperature control in its chat completions reference, but the consumer ChatGPT interface you and your buyers use exposes neither.

On top of sampling variance, ChatGPT layers personalisation. OpenAI's memory FAQ describes how the assistant can carry details from prior conversations into new ones. If you have spent six months discussing your own company in that account, your account is the single worst place to measure whether strangers hear about your company.

There is also a retrieval layer. When ChatGPT searches the web, it pulls live pages and links to them, as OpenAI describes in its announcement of ChatGPT search. Two runs seconds apart can retrieve different pages and therefore reach different conclusions.

The stakes are why this matters more than a curiosity. Pew Research Center found that Google users clicked a result link on 8% of visits where an AI summary appeared, versus 15% of visits without one. When the answer absorbs the click, being named inside the answer becomes the visibility that counts — and you cannot manage it from a screenshot.

Build a prompt set that matches real buyer questions

Your prompt set is the instrument. Get it wrong and every number downstream is noise. The mistake almost everyone makes first is tracking prompts containing their own brand name — “is Growli good?” — which measures nothing, because you have already handed the assistant the answer. Track the questions a buyer asks before they know you exist.

Pull the raw material from places where real language lives: sales call recordings, support tickets, your site search logs, the questions in your onboarding form, and the long-tail queries in Google Search Console. Then shape them into four families.

Aim for 30 to 60 prompts, weighted toward the families that map to your revenue. Write them the way a person types, not the way a marketer writes headings — “best X for small teams under $50/month”, not “X solutions overview”. Then freeze the wording. Every edit resets the baseline, so keep a changelog and treat a reworded prompt as a new prompt with a new history rather than a continuation of the old one.

• Category discovery: “best project management tool for construction firms”, “who does SOC 2 audits for seed-stage startups”.
• Comparison: “X vs Y for enterprise”, “alternatives to X”, “cheaper option than X with SSO”.
• Use-case and job-to-be-done: “how do I migrate 40,000 SKUs to a new PIM”, “tool to track AI search visibility”.
• Objection and risk: “is X secure enough for healthcare data”, “which vendors in this category have bad support”.

Decide the cadence and hold conditions constant

Cadence is a trade-off between statistical confidence and cost. A single run per prompt per week tells you almost nothing, because the variance between runs is larger than most week-over-week movement you care about. A practical floor is three runs per prompt per cycle, weekly, so presence is expressed as a rate — “named in 7 of 12 runs” — rather than a binary.

Increase frequency around events, not permanently: a product launch, a pricing change, a competitor's funding announcement, a wave of new review-site content. Between events, weekly is enough to see direction while keeping the series readable.

Conditions matter as much as frequency. The rule is boring and non-negotiable: change one thing at a time, and never change the measurement setup and the content strategy in the same week.

Also record the model version alongside every run. Model updates are the single largest source of step-changes in AI answers, and a trend line without version annotations will send you chasing content problems that were actually platform changes.

• Run logged out, or in a clean account with memory and custom instructions disabled.
• Fix the locale and language per prompt; track the same question in a second market as a separate prompt, not a variation.
• Use the same surface consistently — the web interface with search enabled behaves differently from a bare API call.
• Timestamp everything in UTC so cycles are comparable.

Record five fields for every run

Presence is the headline, but presence alone cannot tell you what to do next. Log five fields for each run and the diagnosis usually writes itself.

Presence. Was the brand named at all? Aggregate to a share of runs per prompt, then roll up to a share across the prompt set. This is the number you report.

Position. Where in the answer did the mention land — first named option, mid-list, or a footnote after the recommendation? Assistants front-load; being sixth in a list of six is closer to absence than to presence.

Sentiment and framing. Not just positive or negative, but the qualifier attached to you: “good for enterprise but expensive”, “strong reporting, limited integrations”. Those qualifiers are the assistant's compressed summary of everything it has read about you, and they are the most actionable text in the whole dataset.

Competitors named. Log every other brand in the answer. This gives you the assistant's real definition of your category, which is often not the one on your positioning slide, plus a ranked list of who you are actually losing share of answer to.

Citations. Capture every URL the answer links to. This is the causal layer. If a competitor is winning a prompt because a listicle on a third-party site ranks them first, the fix is that page — outreach, a correction, a better data point — not another blog post on your own domain. OpenAI documents its crawlers and how publishers can control them in its bots documentation, which is worth reviewing before you conclude that your own content is invisible for content reasons rather than access reasons.

Read the trend, not the answer

With four or five cycles logged, stop reading individual answers and start reading the series. Three patterns are worth acting on, and one is worth ignoring.

A sustained rise or fall in presence rate across a family of prompts — say, all your comparison prompts — is a real signal, usually traceable to new third-party content entering or leaving the citation pool. A step change across the whole prompt set on the same day is almost always a model update; check your version annotations before you rewrite anything. A change in the qualifier attached to your brand, even with presence flat, is the earliest warning you will get that a narrative is forming, and it is the cheapest to fix.

What to ignore: single-prompt, single-cycle movement. That is sampling noise, and chasing it burns credibility with your leadership faster than reporting no movement at all.

Segment the roll-up by prompt family before you present it. Being strong on use-case prompts and weak on category discovery is a completely different problem from the reverse — the first is a positioning gap, the second is a coverage gap in the places assistants source from. We cover the diagnostic framing in more depth in our guide to AI visibility.

Manual tracking, and where automation earns its place

You can run this manually. Build the prompt set in a spreadsheet, run three passes per prompt in a clean browser session, and log the five fields. For 30 prompts at three runs, that is 90 conversations to read and code per cycle — roughly a day of focused work per week, and the coding is the slow part, because judging position and sentiment consistently across 90 answers is harder than it sounds.

Two things break manual tracking at scale. The first is consistency: two people code sentiment differently, and one person codes it differently on a Friday. The second is coverage — buyers do not only use ChatGPT, and a prompt set measured on one assistant tells you nothing about Gemini, Perplexity, Claude or Copilot, which draw on different retrieval and cite different sources.

This is the problem we built Growli to solve: it runs your prompt set across the major assistants on a schedule, records presence, position, sentiment, competitor co-mentions and citations the same way every cycle, and turns the gaps into a prioritised list of pages and sources to fix. Whichever route you take, the method is the same — fixed prompts, repeated runs, five fields, and decisions made on trends rather than screenshots.

FAQ