If you want to know whether your business is visible in Google AI Mode, ChatGPT or another AI search tool, running a query once is rarely enough.
AI-generated answers can change. A business that appears in one response may disappear from the next. Competitors can change, citations can change and even the way your business is described can vary.
That makes repeated testing useful.
But how many times should you repeat the same query?
There is no universal number of searches that suddenly makes an AI visibility test statistically reliable.
The right number depends on what you are trying to learn.
For a practical manual visibility check, I often use five identical runs. Five runs are not statistically significant and they do not reveal the “true probability” of a business appearing. They simply give us something much more useful than a single answer: an early indication of whether a recommendation looks stable, occasional or absent.
If you want to make stronger statistical claims, you need a different kind of test.
Why One AI Search Is Not Enough
Suppose you ask Google AI Mode:
Which three accountancy firms in Birmingham should a 25-employee manufacturing company using Xero consider?
Your business appears in the answer.
That is interesting.
But what have you actually learned?
You know that your business appeared once.
Run exactly the same query again and the answer may contain three different firms. Run it another three times and you may discover that your business appeared in four of the five responses.
Those two situations are very different:
- appearing once in one search;
- appearing four times across five identical searches.
The first gives you an observation.
The second begins to tell you something about consistency.
That is why repeated testing matters.
Is 30 Searches the Magic Number?
You may encounter advice that says you should run an AI search exactly 30 times because a sample size of 30 is automatically statistically reliable.
That is too simplistic.
Thirty is a familiar number in introductory statistics, but there is no general rule that says:
30 Google AI Mode searches = statistically reliable AI visibility measurement.
Before deciding how many observations are needed, you first need to decide what you are actually trying to measure.
Are you asking:
- Does the answer fluctuate when I repeat the same query?
- How frequently does my brand appear?
- How visible is my brand across several different customer queries?
- Does visibility differ by location?
- Has visibility changed over several months?
- What proportion of future users are likely to see my business?
Those are different questions.
They may require different testing methods.
Simply reaching 30 observations does not resolve that.
Start by Defining the Test
I find it useful to separate AI visibility testing into three broad types.
1. A Quick Repeatability Test
This is the simplest manual test.
Run the same query several times while keeping the conditions as similar as reasonably possible.
For example:
I run a small manufacturing company in Birmingham with 25 employees. I need an accountancy firm to handle year-end accounts, corporation tax, payroll and monthly management accounts. I use Xero and would prefer a firm offering fixed monthly fees. Which three Birmingham accountancy firms should I consider, and why?
Run that query five times.
Do not deliberately change:
- the wording;
- the location;
- the business requirements;
- the device;
- other important variables.
The question you are asking is:
How much does the answer change even though I haven’t deliberately changed the query?
Five runs will not establish statistical significance.
That isn’t the purpose.
It is a practical fluctuation check.
If your business appears:
- 5/5 times, that looks much more stable than a single appearance;
- 3/5 times, it appears to be part of the recommendation set but not consistently;
- 1/5 times, that appearance may have been relatively incidental;
- 0/5 times, you have not observed visibility for that particular query during the test.
Those are observations, not explanations.
2. A Wider Visibility Test
The next question is different:
Does my business appear across the range of questions potential customers might actually ask?
A customer rarely asks only one question.
An accountant, for example, might want to test prompts involving:
- small limited companies;
- manufacturing businesses;
- Xero users;
- property companies;
- monthly management accounts;
- fixed monthly fees;
- an overdrawn director’s loan account.
These should not simply be treated as another 30 repetitions of one experiment.
They are a portfolio of different customer queries.
That allows you to examine your visibility across different needs and commercial situations.
A business might be invisible for a broad query such as:
Best accountants in Birmingham
but highly visible for:
Birmingham accountant for a 25-employee manufacturing company using Xero that needs monthly management accounts.
From a commercial perspective, the second result may be considerably more interesting.
3. Statistical Measurement
A third objective would be to estimate something much stronger, such as:
My business has a 65% probability of appearing for this query.
That is no longer a simple manual visibility check.
You are attempting statistical estimation.
At that point you need to think carefully about:
- the population you are trying to represent;
- the level of precision you require;
- the confidence you want in the estimate;
- whether observations are sufficiently independent;
- whether different locations should be combined;
- whether desktop and mobile results belong in the same sample;
- whether searches over several weeks represent the same conditions;
- whether the AI system itself changed during the test period.
There is nothing wrong with conducting a larger study.
But there is an important difference between:
“Our business appeared in four of five manual test runs.”
and:
“Our business has an 80% probability of appearing.”
The first is a direct description of what you observed.
The second is a statistical claim about a wider population.
Do not confuse the two.
Why Changing Variables Changes the Question
Some AI visibility testing advice recommends varying location, device, browser state and search wording within the same test.
Those variables may absolutely be worth testing.
But once you deliberately change them, you are answering a different question.
Consider location.
Suppose you run 15 searches from Birmingham and another 15 from Manchester.
You may have collected 30 observations.
But you have not repeated the same experiment 30 times.
You have deliberately tested at least two different geographic conditions.
That might be exactly what you want if your question is:
Does my visibility change according to the searcher’s location?
It is less useful if your question is:
Does Google AI Mode give me a consistent answer when I repeat exactly the same customer query under broadly similar conditions?
Neither experiment is inherently better.
They simply measure different things.
A Simple Five-Run Manual Test
For businesses that want a practical starting point, this is the method I use.
Step 1: Choose a Meaningful Customer Query
Do not start with your business name.
Choose something a genuine potential customer might ask when looking for a solution.
Detailed queries are particularly useful because they allow you to test whether AI systems understand when your business is a strong fit for a specific requirement.
Step 2: Freeze the Query
Write down the exact wording.
Use that same wording for every run.
Do not make small improvements between searches.
If you change the prompt, you have changed one of the variables.
Step 3: Run It Five Times
Run the identical query five times.
The purpose is not to create a statistically definitive sample.
It is to move beyond the false certainty created by looking at a single AI response.
Step 4: Record the Results
For each run, record at least:
| Metric | What to record |
|---|---|
| Presence | Did your business appear? |
| Prominence | Where did it appear in the answer? |
| Portrayal | How was the business described? |
| Competitors | Which other businesses appeared? |
| Citation | Was your website or another source cited? |
| Source URL | Which page supported the recommendation? |
| Date | When was the test conducted? |
You can add more detail where useful, but these fields already tell you considerably more than a simple yes/no visibility check.
Step 5: Look for Patterns
You might find:
| Business | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Presence |
|---|---|---|---|---|---|---|
| Business A | Yes | Yes | Yes | Yes | Yes | 5/5 |
| Business B | Yes | No | Yes | No | Yes | 3/5 |
| Business C | No | No | Yes | No | No | 1/5 |
That immediately gives a richer picture than one search.
Business A appears stable within this small test.
Business B appears regularly but not consistently.
Business C appeared once.
That still does not tell you why.
Separate Measurement From Explanation
This is one of the most important principles in AI visibility testing.
Suppose your company appeared in only one of five searches.
You can say:
My company appeared in one of the five test runs.
You cannot automatically conclude:
Google doesn’t trust my website.
Perhaps:
- another business matched the query more closely;
- several competitors were being rotated;
- third-party sources affected the answer;
- geographic factors mattered;
- the AI output simply fluctuated;
- your website lacked information relevant to that particular requirement.
Visibility testing tells you what happened.
Explaining why it happened often requires further investigation.
The same applies at the other end of the scale.
Appearing five times out of five does not prove that your business is an “authority node” or that Google considers it the market leader.
It tells you something much more specific:
Your business appeared in all five observations for that query during that test.
That is useful evidence without pretending it proves more than it does.
Presence Is Only One Part of AI Visibility
Counting appearances is useful, but it is not the whole story.
Imagine two businesses both appear in five out of five searches.
Business A is consistently presented first and described as particularly suitable for the customer’s requirements.
Business B appears near the bottom of each answer with a generic description.
Their presence rate is identical.
Their visibility is not.
That is why AI visibility testing should consider more than simple mentions.
Look at:
Presence
Does the business appear at all?
Prominence
How visible is it within the answer?
Portrayal
What does the AI actually say about it?
Persuasion
Does the answer give the customer a reason to choose it?
A raw percentage can hide all of that.
Test Commercially Meaningful Queries
There is another reason not to become obsessed with sample size.
You can run a query 100 times and still be measuring something with very little commercial value.
For example:
accountants Birmingham
may tell an accountancy firm something about broad visibility.
But:
I run a 25-employee manufacturing company in Birmingham, use Xero and need payroll, year-end accounts and monthly management accounts on a fixed monthly fee
reveals whether the firm appears when a potential customer has a much clearer requirement.
The second query may be searched less often.
But a recommendation at that stage of the customer journey could be much more commercially important.
The quality of the query therefore matters as much as the number of repetitions.
When Should You Run More Than Five Tests?
Five runs are a starting point, not a rule.
Run more tests when the question you are trying to answer requires more evidence.
For example, additional testing may be useful if:
- the first five results are highly variable;
- you want to compare visibility over time;
- several locations matter to the business;
- you want to compare desktop and mobile behaviour;
- you are tracking a large portfolio of valuable customer queries;
- the commercial importance of the result justifies more rigorous measurement.
At that point, automated AI visibility monitoring may become more practical than repeatedly carrying out manual searches.
The important principle remains the same:
Choose the method according to the question you are trying to answer.
Do not choose a number first and design the experiment around it afterwards.
A Practical Starting Framework
For a small business manually testing AI visibility, I would start like this:
- Choose one commercially meaningful query.
- Write down the exact wording.
- Run it five times without deliberately changing the main conditions.
- Record presence, prominence, portrayal, competitors and citations.
- Look for repeated patterns rather than judging the first response.
- Repeat the process for other important customer queries.
- Revisit the same tests later if you want to monitor change over time.
That will not give you a statistically definitive probability of appearing.
It will give you something much more useful than a single search:
evidence of how your visibility actually behaves.
The Main Point
There is no magic number of AI searches that automatically creates a reliable visibility test.
Five runs are useful for a practical repeatability check.
Thirty runs may provide more observations, but thirty does not automatically make an experiment statistically significant — particularly if you are changing devices, locations, prompts and other conditions at the same time.
Larger statistical studies require a properly defined question and an appropriately designed sample.
For most businesses beginning to test AI visibility manually, the best first step is simpler:
Repeat the same commercially meaningful query several times, record what actually happens and avoid claiming more than the evidence shows.
That is the foundation on which more sophisticated AI visibility measurement can be built.