Guest Column: Pros and Cons of User Reviews
With the growth of consumer product information online, user generated content (UGC)—especially user reviews—has become increasingly important and valued in consumer buying decisions. Product user reviews on sites such as Amazon, PriceGrabber, Power Reviews, Walmart, CNET, Google Product Search, and our own site, ConsumerReports.org, are popular tools for helping consumers make purchase decisions. Would-be buyers seek information from peers, and then share their own experiences back through these and other websites after making a purchase. This facilitates a productive dialog between experts, professionals, publishers, and others on consumer products.
UGC empowers consumers in many ways. It provides information about others' experiences with products, offers warnings about dangerous products or poor services, and holds the potential for changing the marketplace when companies take notice. With a simple click of the mouse, consumers can tap into the collective intelligence of a worldwide community.
There is much to be learned from the experiences others have with the products that they have purchased, but using this information requires that consumers understand the context of review experiences and motivations of the reviewer in order to help distinguish good advice from bad or irrelevant counsel. How, for example, do you know that the reviewer really owns the product, and is not a shill for the company? How can you tell whether he knows what he's talking about? And how do you sort through hundreds of reviews and individual experiences to find the ones that will provide the best information for you?
The challenge here is recognizing that there may be a gap between your needs and the information conveyed in the review. The differences may be significant or apply only in certain situations. And what about the experts? Where do they fit in? How do consumers define "expert"? "Mommy bloggers" are often considered a trusted source of information by other moms, but does that make them truly experts—especially since some may be receiving compensation of some sort from the companies they write about? What do experts really know and how does it apply to your needs?
Combining Expert and Consumer-Driven Reviews
Since Consumer Reports is widely known for its laboratory tests of new products, you might assume that we would discount the importance of user reviews. Nothing could be further from the truth. Sure, laboratory tests are the best source for objective, performance-related information about a product—i.e., how it compares to other models, how it holds up, etc. And, we have relied on significant consumer survey research for decades to capture user experiences regarding product satisfaction, product reliability, repair history, etc. These surveys use statistically valid sampling techniques to be able to reliably project the results to a larger population.
But user reviews can provide valuable information. They capture subjective information about the product based on the individual's experiences and preferences. User reviews are particularly useful for things like appearance and ease of use. We believe that combining both objective and subjective data can very helpful to the consumer. We at Consumer Reports are working to effectively combine our expert product testing and research results with user generated product reviews. Our hope is to identify and clarify any discrepancies between the two and thus encourage a dialogue involving experts, product owners and would-be buyers.
Considering the Source
Today, you can find user reviews for just about every consumer product on just about any site that reports on these products or sells them on the Internet. Most sites allow consumers to rate products on a scale of 1 to 5. Others have begun to incorporate added features to provide more robust information. For example, Amazon allows users to post photos of the product. Walmart offers some basic demographic information about the user (age, gender) and allows users to rate different attributes (value, features, appearance, etc.), as well as provide an overall rating. Others ask the user to answer specific questions such as its pros, cons, best uses, and so forth. Indeed, so many sources exist for user reviews today that some experts say their ubiquity has made them a commodity.
User reviews are also widely known to be prone to certain kinds of bias. For example, when you ask people to rate products on a five-point scale, the largest share of products will get a relatively high rating of 4. Studies have shown this; it's just human nature. So the consumer is left with the challenge of how to analyze hundreds of user reviews to find the product that's best.
A recent example may help explain why this is so important. Consumer Reports regularly tests vacuum cleaners in our labs, and so took note when roughly half of the user reviews for two of our most highly-rated vacuums complained about durability problems. We paid attention to the consumers' experience and vowed to follow-up with additional research. When we did—by re-contacting over 1,000 product owners who had answered our questionnaire—our results confirmed that the two models were, in fact, relatively reliable. Even the best vacuums have some durability issues. The results of our investigation allowed us to explain how scientific sampling and side-by-side comparison can provide different results from individual experience with unique purchases. Better still, we addressed the consumers' immediate need, and offered guidance on repairs and alternatives. In the end, consumers were heard and engaged, and our understanding of these models expanded.
The illustration below shows the wide array of disparate data elements that consumers often wade through to make purchase decisions. By applying processes such as sentiment analysis (which can determine if raw, unfiltered data is positive or negative in tone) and aggregation of individual experience-based product reviews combined with expert testing and research, we may help consumers make better informed purchase decisions.
Here are seven important digital capabilities that can help consumers make better purchase decisions:
1. Offering full access to individual user reviews for each specific model product, including both structured and non- structured comments.
2. Allowing visitors to "rate" the reviews as to which are most useful. This helps bring the best reviews to the top of the queue, and eliminates reviews that are fake, misleading, or just not particularly helpful.
3. Presenting not only the results from their own reviews, but also aggregate reviews from other sources in order to paint a broader picture of sentiment about the product on the Web.
4. Delivering a real-time automated summary using a data mining approach such as sentiment analysis to report whether the current aggregate consumer experience with a product is positive or negative.
5. Providing expert testing and research findings—e.g., from laboratory testing—in addition to the user reviews.
6. Detecting and highlighting any discrepancies between user experience and expert results. If the users like a product that the experts don't, or vice versa, then try to define the reason.
7. Allowing users to engage in a dialogue with the experts and other users to understand the reason for review discrepancies and the appropriate fit of a product that matches the consumer's specific needs or interests.
We think this approach presents a better solution for consumers because it does two things: Combines user reviews with expert testing, and increases the relevance of data from social media with automated analysis techniques. If these guidelines were followed, it would help consumers better navigate a crowded marketplace, create more feedback and dialogue, and generate insights from real-time user experience that would help buyers and manufacturers alike.
So our long-term goal is to achieve the optimal blend of expertise, technology and features that will help consumers make better choices. PE
John Sateja is the executive vice president for Consumers Union (CU), the nonprofit publisher of Consumer Reports. He oversees publishing, editorial, laboratory testing and consumer research for all Consumer Reports information products including: Consumer Reports magazine, ConsumerReports.org, Consumer Reports on Health, Consumer Reports Money Adviser, Consumer Reports Special Publications, Consumer Reports Auto Price Services, ShopSmart magazine, ShopSmart Mobile, as well as all new product development.