Summary:
AI-generated review summaries extract qualitative key themes from customer feedback, helping shoppers quickly assess what purchasers think about the product.
When designed well, AI summaries of ecommerce reviews helped our study participants quickly gauge product quality and fit for their needs. But when AI summaries of customer reviews were vague, poorly formatted, or blocked actual reviews, they wasted time. To truly improve upon the traditional review-reading experience, designers should leverage what AI is good at without forgetting why customers rely on reviews in the first place.
Traditional Summarization of Customer Reviews
In our decades of ecommerce research — including the past few years with AI features emerging on the scene — reviews have always played a major role in customer decision making. But the volume of reviews has presented a design challenge. While customers rely heavily on reviews, they typically don’t want to read dozens of them. When there are hundreds or thousands of reviews for a single product, it’s difficult for customers to get a quick understanding of reviewers’ conclusions.
Over the years, ecommerce designers introduced clever tactics to give users a quantitative sense of how other customers rated products. To help people navigate an ocean of opinions, retailers have implemented features like:
- Rating averages to give a sense of the overall positivity or negativity of a set of reviews
- Histograms, with the number of reviews for each star category, so customers can see the distribution
- Review filters that allow users to narrow down reviews by like height, size, hair type.
- Keyword searches of reviews
- Representations of themes, keywords, or answers to structured questions (like, Would you recommend this product?)
While certainly useful, these traditional approaches help customers understand the nuance of reviews — which are inherently qualitative. This is where generative AI offers a new approach to review summarization.
AI-Generated Summaries of Reviews
AI-generated summaries of reviews are the latest in a long line of attempts to make large numbers of reviews easy to consume.
An AI summary of reviews aggregates a product’s customer reviews to produce a high-level, qualitative overview of all user feedback.
In our research, participants generally appreciated this simple, yet useful AI feature. For example, one participant told us:
“I just like that this summarizes 1600 ratings. It makes it easier to gauge what people are saying without having to go through every single comment.”
AI-generated summaries of reviews were a well-received feature because they:
- Didn’t require any input from users
- Took advantage of one of genAI’s primary strengths (text summarization)
- Supplemented, but did not replace, traditional review-summarization UI elements such as review average ratings, histograms, or filters
Note that AI summaries don’t eliminate users’ desire to explore the actual individual reviews. For example, one participant told us:
“I’ve noticed that a lot of sites are doing an AI summary of all the reviews. That’s helpful, but I also still take the time to look at reviews. […] It’s useful for identifying common trends, but I still want to see photos and real-life examples of the product in use.”
Additionally, high-level summaries generally won’t answer users’ specific questions the way that individual reviews can. For example, one participant was shopping on Amazon for hair products that would work with her hair type. She was annoyed to see the AI summary focused on the smell and price of the product, but not the consistency. When she scanned through the individual reviews, she was able to find some reviews commenting on product consistency, which answered her question.
The Anatomy of AI-Generated Summaries of Reviews
In our study, AI-generated review summaries were consistently located in the Reviews section on product-description pages right above the individual reviews.
Most AI summaries included a combination of the following elements:
- Label: A section header such as Customers Say or Customers Are Saying
- Summary: Often presented as a paragraph with full sentences and minimal formatting
- Disclaimer: A small note or tooltip explaining that the content is AI-generated and unverified by the company (probably aiming to avoid potential lawsuits if people make purchases based on inaccurate information)
- Themes: A collection of the most-mentioned themes across all reviews (regardless of whether those keywords were mentioned verbatim in the reviews)
Most AI summaries were (unfortunately) presented as blocky, unformatted chunks of text. Lack of formatting made them a bit difficult to scan, but no more so than typical user-generated reviews. The blocks often began with some variation of:
- Customers find value in…
- Customers report…
- Reviewers describe…
The summaries then tended to list out the positive aspects mentioned by reviewers first, followed by any negative aspects or mixed opinions.
Building Trust in AI Summaries
While many participants found the summaries valuable, trust was a major obstacle to their adoption and use. And when it comes to generative AI, some of that caution is well-founded. During our study, we observed several odd issues in AI-generated summaries of reviews. Sometimes, critical details were left out of the summaries, or the summaries included contradictory themes. These were simply the artifacts of the current generative AI technology.
Building trust in AI summaries will require more than a simple Powered by AI tag or disclaimer (though you should absolutely include one that’s large and salient enough to be noticed). It’s also a good idea to:
- Expose the negative themes and perspectives from reviews in the summary
- Supply specific themes that are contextually relevant to the product and your audience
- Cite the sources — count, reference, quote, and link to actual reviews
Expose the Negatives
In the study, users repeatedly said that they expected the AI summary to overemphasize positives while downplaying negatives.
“Sometimes I wonder if it’s a completely accurate summary. […] It’s a good starting point, but I’m not going to put all my trust in it. I’m not necessarily saying the company’s weighting it, but I know in general AI isn’t always accurate, so I want to read a bunch of the reviews myself.”
This concern echoes skepticism we’ve seen in earlier ecommerce studies. Especially on small direct-to-consumer websites with very consistently high average ratings, participants wondered aloud if the site had cherry-picked only the positive reviews. Even a small amount of negativity in reviews helps customers feel that they’re seeing a real depiction of actual experiences.
Avoid the temptation to downplay or hide negative themes in order to increase sales. Presenting potential drawbacks honestly helps build trust and create a better decision-making environment for users.
Supply Specifics
Participants complained when the AI review summaries were too vague or irrelevant to the product type. One woman mentioned that she didn’t like it when they were “too general.” She told us, “That’s like when someone writes a review and just writes, ‘good.’ That doesn’t tell me anything!”
Ideally, the AI system should consider the product category and the product factors evaluated in reviews when generating its summary. The AI-generated review summaries of two different products should look and read differently.
On Dick’s Sporting Goods’ website, the AI summaries were appropriately tailored to the product category and context. For example, this summary of reviews for a driver (golf club) was relevant and used terminology likely to resonate with golfers.
Cite Your Sources
We’ve established that an AI summary can supplement, but not replace, the value of reading actual reviews. Making it easier for customers to “check the sources” of AI summaries can build trust.
In the Etsy example above, not only were the AI-generated themes unhelpfully vague and unspecific to the product, but, even worse, there was no “evidence” that those themes existed in the real reviews.
Amazon’s approach worked much better. The AI-identified themes were clickable. When clicked, each theme expanded to reveal:
- The total number of reviewers who mentioned the theme (264 customers mention “Comfort”)
- The number of reviewers who wrote about the theme positively (247 positive)
- The number of reviewers who wrote about the theme negatively (17 negative)
- A one-sentence summary of the comments related to that theme (Customers find these sandals comfortable and supportive)
- A handful of relevant excerpts from actual reviews, with the relevant portion bolded (“so comfortable that it’s almost like I’m walking barefoot.”)
- Links to the full reviews (Read more >)
This produced a well-balanced, nuanced, and convenient way for participants to investigate product qualities that they found particularly important.
Amazon’s design wasn’t perfect. One serious issue was that, despite being styled like links and having a Select to learn more label, our participants didn’t expect those themes to be interactive. For example, one participant even pointed this section out, mentioning that she liked the themes being listed there. But she didn’t realize they were clickable until a study facilitator asked her to expand one. Discoverability is a common problem for AI features.
Conclusion
Ecommerce UX is all about helping customers make the right decisions for them — building positive long-term relationships that keep them coming back, while preventing disappointment and unnecessary returns as much as possible.
AI-generated summaries of customer reviews should complement — not replace — traditional review summarization techniques, providing a quick overview of the reviews, while still allowing userss to explore individual reviews for specific details.
To maximize effectiveness, AI summaries should:
- Present both positive and negative themes to build trust
- Offer product-specific insights rather than generic statements
- Connect summaries to actual reviews through counts, quotes, and links
- Ensure that interactive elements are clearly discoverable
Before adding AI summaries to your site, carefully consider whether they bring enough improvement to justify the additional space and interface complexity. The most successful implementations leverage AI’s summarization strengths while preserving what customers value most about traditional reviews: authentic experiences from real users that answer their specific questions about products.