What Does a High-Quality AI Recommendation Actually Look Like? Testing “Mention Quality” in RAMP

Being recommended by an AI system sounds like a good result for a business.

But what exactly was the customer told about that business?

An AI answer might simply provide a name. It might explain convincingly why the business fits the customer’s needs. Or it might produce a highly detailed, persuasive recommendation where some of that apparent specificity is less firmly supported than it first appears.

That difference is what M — Mention Quality in our RAMP Framework is intended to explore.

To test it, we created a realistic customer situation involving a Victorian terraced house in Bristol and asked Google AI Mode to recommend three architects.

We ran the identical prompt five times.

The recommendations were consistently detailed and closely matched to the customer’s requirements. But when we checked some of the claims afterwards, several different patterns emerged.

Some highly specific recommendations were strongly supported by information the business itself had published.

Some had a sound factual core, but AI added further interpretation or specificity.

In one case, general information about Bristol architects appeared to become more specific to the recommended firm than the underlying page clearly supported.

And in others, the businesses appeared highly relevant to the project while their own positioning did not neatly match the professional category the customer had requested.

That gives us a more useful way of thinking about Mention Quality than simply asking whether an AI recommendation sounds positive.


What Is Mention Quality?

RAMP considers AI visibility through four connected questions:

  • R — Relevance: Was this a commercially useful customer search to appear for?
  • A — Appearance: Did the business appear?
  • M — Mention Quality: How was the business presented?
  • P — Pathway: Was there a practical route from the answer towards the business?

Our previous Pathway experiment looked at what happened after a recommendation: could the customer actually move towards the business?

This experiment concentrated on the recommendation itself.

We wanted to know:

Does AI merely name a business, or does it explain why that particular business fits that particular customer’s needs?

And, importantly:

How well does the evidence support that explanation?


The Customer Scenario

We used the following prompt:

I own a Victorian terraced house in Bristol and am planning a loft conversion to create an extra bedroom and bathroom. I need an architect who has experience with loft conversions and older residential properties, can handle the planning application and Building Regulations drawings, and can help manage the project through the construction stage if needed. I would prefer a firm that clearly explains its process and fees. Which three Bristol architects should I consider, and why?

This was deliberately more specific than simply asking for the “best architects in Bristol”.

The customer wanted:

  • loft-conversion experience;
  • older residential property experience;
  • planning support;
  • Building Regulations drawings;
  • construction-stage assistance;
  • a clear process;
  • and transparent fees.

That gave Google AI Mode several distinct requirements against which to explain each recommendation.


How We Tested It

We ran the identical prompt five times in Google AI Mode.

We recorded the recommendations before independently checking the businesses.

That produced 15 recommendation instances:

BusinessAppearances
Wall Architecture5
Clifton Design3
DHV Architects2
Minton Architects2
3bd Architects1
Adapt Planning1
Excellence Living1

Wall Architecture appearing in all five runs was interesting in itself.

But our main question was not simply:

Who appeared most often?

It was:

What exactly was the potential customer told about each business?


What We Found

In these five runs, Google rarely gave a generic recommendation.

Instead, it repeatedly tried to connect individual firms to specific parts of the customer brief.

The answers discussed subjects including:

  • Victorian houses;
  • loft conversions;
  • difficult staircase layouts;
  • headroom;
  • conservation areas;
  • planning applications;
  • Permitted Development;
  • Building Regulations;
  • structural engineers;
  • construction-stage support;
  • fee structures;
  • and project timescales.

On the surface, this looked like excellent Mention Quality.

But the verification stage showed that there was a spectrum between specificity that was strongly supported and specificity that sounded more firm-specific than the available evidence clearly justified.


Clifton Design: What Strong Mention Quality Can Look Like

Clifton Design provided perhaps the clearest positive example.

Across its appearances, AI Mode gave remarkably precise information.

It referred to:

  • written quotations within five working days;
  • planning drawings within two to three weeks;
  • Building Regulations drawings within three to four weeks;
  • a £500 planning-stage fee;
  • a £350 Building Regulations-stage fee;
  • a £1,100 rear-dormer loft conversion fee;
  • and an additional £150 for a Certificate of Lawfulness in the example given.

Those claims initially looked almost unusually specific for an AI recommendation.

But Clifton Design’s own published process information substantially supported them.

That matters because the customer explicitly asked for a firm that clearly explained its process and fees.

AI did not merely call Clifton “transparent”.

It surfaced concrete information showing why it regarded the firm as a good match.

That is a strong example of Mention Quality:

The recommendation was specific, relevant to the customer’s requirements and supported by information published by the business itself.

There is also a useful practical lesson here.

Publishing clear information does not guarantee that an AI system will recommend a business.

But if the information exists, an AI system has firmer evidence from which it may potentially explain why the business is relevant.


Wall Architecture: Consistent Portrayal, Changing Detail

Wall Architecture appeared in all five runs.

The exact explanations changed.

Different answers emphasised:

  • Victorian-property constraints;
  • loft conversions;
  • staircase positioning;
  • roof headroom;
  • planning;
  • Building Regulations;
  • structural coordination;
  • RIBA project stages;
  • tendering;
  • fees;
  • and construction support.

But the underlying portrayal remained broadly consistent.

Google repeatedly presented Wall Architecture as a Bristol residential practice suited to a technically challenging period-property project and capable of supporting the customer beyond the initial design stage.

That suggests an important distinction:

Mention consistency does not require wording consistency.

The same business can be described differently on repeated searches while retaining a broadly stable customer-facing identity.

However, Wall also showed another side of Mention Quality.

Its published information strongly supports the core recommendation: residential architecture, planning, Building Regulations and later-stage project support.

Some AI descriptions went further, introducing very specific references to Victorian loft issues such as staircase configuration and headroom optimisation.

Those are highly relevant considerations.

But the firm-specific support for every one of those details was not always as obvious as the support for the underlying services.

So a recommendation can be fundamentally well grounded while AI adds additional detail that makes the match sound even more precise.

That detail is not necessarily wrong.

But specificity by itself should not be mistaken for verification.


DHV Architects: Evidence Plus AI Judgement

DHV Architects gave us a similar but slightly different example.

There was strong underlying evidence for recommending the practice.

DHV describes relevant residential and historic-building conservation work, and there was published project evidence involving Victorian properties and loft conversions.

But AI Mode went beyond describing those facts.

It used phrases such as:

  • “renowned period property specialists”;
  • “outstanding track record”;
  • and “premium choice”.

Those are judgements.

That does not automatically make them inappropriate. Recommendations inevitably require some synthesis and interpretation.

In fact, synthesis is part of what makes an AI recommendation useful.

The important question is whether the strength of the conclusion remains proportionate to the evidence.

There is a difference between:

This practice undertakes historic-building conservation and relevant residential projects.

and:

This is the premium choice for your project.

One is primarily factual.

The other is AI’s evaluation of those facts.

A good Mention Quality assessment needs to recognise both.


Minton Architects: When General Information Starts to Sound Firm-Specific

Minton Architects produced the most revealing result in the experiment.

Google recommended the firm twice and described it in highly specific terms.

Among other things, the answers discussed architectural fees of around 7% to 14% of construction cost, or fixed packages of approximately £1,500 to £4,000+ VAT.

That made it sound as though Minton itself had published those figures as its pricing for this type of Bristol project.

The underlying Bristol page was more nuanced.

It discussed what architects in Bristol generally do and what architectural fees for Bristol loft conversions typically cost.

That is useful information.

But it is not necessarily the same as saying:

These are Minton Architects’ own fees.

This gives us what might be described as a firm-specificity gap:

Information may be relevant to the subject, but the AI recommendation can make it sound more specifically attributable to the recommended business than the underlying material clearly supports.

That distinction is important because the resulting recommendation sounds exceptionally well matched to the customer.

It contains the type of information the customer explicitly requested.

But the apparent firm-specificity is stronger than the source itself necessarily warrants.


Adapt Planning and Excellence Living: Relevant to the Need, But What About the Category?

Two recommendations raised a different Mention Quality question.

The customer specifically said:

“I need an architect…”

Adapt Planning was recommended once.

Its services were clearly relevant. It provides architectural design and planning services, has loft-conversion material and Bristol project experience.

However, its own positioning includes describing its service as an alternative to traditional architects.

That does not mean it could not be an excellent option for the customer’s project.

It does raise a narrower question:

Does the business neatly fit the professional category the customer explicitly requested?

Excellence Living made this issue even clearer.

Again, it appears highly relevant to the actual project.

Its website discusses:

  • loft conversions;
  • planning and design;
  • structural engineering;
  • and project delivery.

But its own positioning is more strongly associated with property renovation and a contractor-led design-and-build approach.

Google nevertheless presented it alongside Wall Architecture and Clifton Design in response to a request for three Bristol architects.

The point is not that Excellence Living was necessarily a poor recommendation.

It may well be commercially useful to the customer.

The Mention Quality question is more precise:

Has AI accurately characterised the type of business it is recommending?

A company can be very well suited to solving the customer’s problem without necessarily matching the exact professional category requested.


The Problem Is Not AI Synthesis

It would be easy to interpret examples like these as evidence that AI should simply repeat facts from business websites.

I don’t think that is the right conclusion.

Synthesis is what makes an AI recommendation useful.

A customer does not necessarily want ten disconnected facts.

They want help answering:

Which of these businesses seems most appropriate for what I need?

AI therefore has to interpret information.

Clifton Design demonstrates how powerful that can be when clearly published evidence is turned into a customer-specific explanation.

The quality issue arises when that synthesis becomes more specific, certain or firm-specific than the evidence reasonably supports.

That is a much more useful distinction than saying AI should never infer anything.


Six Questions for Assessing Mention Quality

This experiment suggests six practical questions that can help assess M — Mention Quality.

They are not intended as an established scoring system.

They are simply questions that help us investigate what an AI recommendation is actually communicating.

1. Specificity

Does AI merely name the business, or does it explain why the business is being recommended?

A specific explanation can be much more useful than a simple appearance.


2. Customer Matching

Does the explanation address the customer’s actual requirements?

In this experiment that included:

  • Victorian-property experience;
  • loft conversions;
  • planning;
  • Building Regulations;
  • construction support;
  • process;
  • and fees.

A detailed answer is of limited value if the detail does not relate to what the customer needs.


3. Supportability

Can the important claims actually be substantiated?

Clifton Design provided a particularly good example of detailed claims that could be traced back to published information.


4. Firm Specificity

Does the evidence genuinely describe this particular business?

Or is AI taking general industry information and presenting it as though it describes the recommended firm specifically?

The Minton example shows why this matters.


5. Consistency

When the same business appears repeatedly, is its overall portrayal reasonably stable?

Wall Architecture showed that wording and individual details can change while the broader interpretation of the business remains consistent.


6. Professional or Business Category Fit

Does the recommended company actually fit the type of provider the customer requested?

A business can be relevant to solving the problem while its own positioning may not neatly match the category used in the AI answer.

Adapt Planning and Excellence Living highlighted this question.


Why We Are Not Turning Mention Quality Into a Score

It would be easy to give each of these six factors a rating and create a percentage.

For example:

Mention Quality: 82%.

But what would that really mean?

Why should Supportability carry the same weight as Consistency?

Should Category Fit matter more than Specificity?

Would the weighting be the same for an architect as for a restaurant, dentist or software provider?

We don’t currently have evidence to justify such a formula.

For now, Mention Quality seems more useful as a diagnostic framework.

Instead of simply recording:

We appeared three times.

we can ask:

What exactly was the customer told about us on those three occasions?

That may reveal considerably more.


The Same Appearance Rate Can Hide Very Different Visibility

Imagine two businesses each appear in three out of five tests.

Their basic Appearance rate is identical:

3/5.

But suppose one is repeatedly described using:

  • relevant firm-specific evidence;
  • accurate services;
  • transparent published information;
  • and a reasonably consistent portrayal.

The other is described using:

  • generic market information;
  • stronger claims than the evidence clearly supports;
  • changing interpretations;
  • or a questionable description of the company’s actual category.

Both businesses appeared equally often.

But the quality of that visibility is not necessarily equivalent.

This is exactly why RAMP separates Appearance from Mention Quality.


What Businesses Can Take From This

Businesses cannot dictate exactly how Google AI Mode, ChatGPT or another AI system will describe them.

Nor can publishing particular information guarantee that the business will be surfaced or recommended.

But businesses can control how clearly they explain themselves.

If customers routinely need to understand:

  • what type of work you specialise in;
  • which customers you serve;
  • what problems you solve;
  • what your process involves;
  • which services are included;
  • how fees work;
  • what restrictions apply;
  • what happens at the next stage;
  • or what evidence you have of relevant experience,

there is a strong case for addressing those questions clearly.

This connects directly with our work on choosing AI visibility prompts around real customer needs.

A generic query such as:

“Who are the best architects in Bristol?”

would tell us much less about how AI understands each business.

Our detailed customer situation gave Google specific requirements to respond to.

That, in turn, gave us much more to test.


What This Experiment Does — and Does Not — Show

This was a small exploratory experiment involving:

  • one customer situation;
  • one sector;
  • one location;
  • Google AI Mode;
  • five repeated searches;
  • and 15 recommendation instances.

It does not establish how AI Mode behaves across every business category or every query.

Nor can the sources displayed with an AI response be assumed to reveal Google’s complete internal synthesis process.

What we can do is examine what the customer was shown and compare important claims with appropriate published information afterwards.

In these five runs, AI Mode was very good at producing recommendations that felt closely tailored to the customer’s requirements.

Verification then revealed a range of outcomes:

Strongly supported customer matching — Clifton Design was the clearest example.

A supported core with additional AI interpretation or specificity — Wall Architecture and DHV illustrated this.

A firm-specificity gap — Minton provided the strongest example.

A professional-category fit question — Adapt Planning and Excellence Living raised this issue.

That spectrum may tell us more about Mention Quality than simply counting how many times each business appeared.


Final Thought

We started with a simple question:

What does a high-quality AI recommendation actually look like?

After these five runs, I don’t think the answer is:

The most detailed recommendation.

Detail is useful.

Customer matching is useful.

AI synthesis is useful.

But a genuinely strong mention should also be appropriately supported, genuinely connected to the particular business being recommended, reasonably consistent and accurate about what that business actually is.

That leads to a more useful definition of M — Mention Quality:

Mention Quality is not simply how positive or detailed an AI recommendation sounds. It is how accurately, specifically and consistently the AI explains why a particular business fits the customer’s need, and how well that explanation is supported by evidence genuinely relating to that business.

Being recommended is worth measuring.

But once the business appears, there is another question worth asking:

What exactly did AI tell the customer about us — and how well does that portrayal stand up when we check it?

Leave a Comment