How to ethically improve your customer reviews

By Felix Salmon
May 23, 2011
David Segal's Haggler column this week concentrated on companies which use Mechanical Turk to plant fake reviews -- both positive and negative -- on Yelp.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

David Segal’s Haggler column this week concentrated on companies which use Mechanical Turk to plant fake reviews — both positive and negative — on Yelp. “This is very sneaky,” he writes, “and it’s a continuing problem for Yelp, which is locked in a “Spy vs. Spy”-style contest with fake reviewers.”

But Panos Ipeirotis has found a much more interesting way of using Mechanical Turk to game Yelp; it may or may not be sneaky, but it does seem to give companies a significant advantage — enough that Zappos seems to have spent roughly half a million dollars on it. It’s worth noting that both Zappos and Mechanical Turk are owned by Amazon, so if Yelp has an issue with this, it’s going to be fighting a very large opponent.

The insight here is that the kind of companies Segal is writing about are looking at Yelp in a pretty naive way: good reviews are good, bad reviews are bad. But in fact it’s more subtle than that. In a recent paper, Ipeirotis looked at a long list of variables in hotel reviews, to see which ones were good predictors of customers actually booking those hotels. The statistically-significant variables were these:

statsig.tiff

Top of the list — the thing you really want if you want your hotel to get booked — is “beach.” But hotels can’t do much about their proximity to the beach. What they can do is address the second-most important variable: readability. Just having well-written reviews, it turns out, is much more important than having good reviews: the rating given in the review was much less significant, as were aspects of the review relating to cleanliness, check-in, service, and the like. And the Value rating on reviews actually had a significantly negative correlation: the better value reviewers said a hotel was, the less likely it was to be booked.

This holds true for other products, too, as Ipeirotis found in another paper:

Demand for a hotel increases if the reviews on TripAdvisor and Travelocity are well-written, without spelling errors; this holds no matter if the review is positive or negative. In our TKDE paper “Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics”, we observed similar trends for products sold and reviewed on Amazon.com.

So Zappo’s, instead of getting people to write good reviews, just got them to fix reviews which already existed — deal with spelling errors, correct grammar, that kind of thing. And anecdotally (Ipeirotis got to talking to someone over drinks at WWW India), Zappo’s saw a “substantial” improvement as a result of its investment in cleaning up such things.

Is it fair for Zappo’s to give itself an advantage by doing its reviewers a favor and making them seem more literate than they actually are? I’m not sure. But I’m thinking that I should start correcting spelling and grammar mistakes in my comments section. It’ll only serve to improve what people think of it — and of my blog more generally. Even if those people are very rude about me.

(H/T Hanson)

8 comments

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/

it wud b mor ethikal f they says reviews waz editd n provaid lnk to orignal txt.

Posted by guanix | Report as abusive

Well, you might want to start with the spelling in your post. I’m pretty sure is is Zappos not Zappo’s. While the title of your post is how to ETHICALLY improve customer base, I think that the implication of the research of the reviewer bots on Yelp is that they should mention that a crime happened. Seems like the impact is almost as negative as spelling is positive.

Also not mentioned is that the number of reviews are negative? I find this odd because I myself use number of reviews as a positive indicator when making purchases on Amazon, Expedia, etc.

Posted by chappy8 | Report as abusive

Chappy8, you should check your own grammar before criticizing others.

Posted by Frostslasher | Report as abusive

I can understand the correlation with review value. I know that when I search through reviews on, for instance, a hotel, I usually only scan through the reviews for the negatives. They’re generally much more informative to me about the hotel than 100 reviews saying “We had a great stay!”. They also tend to spell out to you what the clientele there is like. If the negative reviews are “Poor caviar selection from room service” as opposed to “Roaches in the pillows”, you can make some educated guesses about what the other guests will probably be like.

Posted by spectre855 | Report as abusive

So how does Matt Yglesias still have a job?

Posted by RZ0 | Report as abusive

RZ0 – It was Ezra, not Matt, who was hired by Kaplan.

Posted by klhoughton | Report as abusive

spectre855: I see what you are saying. I was thinking ‘all else equal’. If I were comparing two places that got 3 stars, I would pick the one that had more reviews. Clearly your thinking is more prevalent since the number of overall reviews is negatively correlated to actual bookings.

Posted by chappy8 | Report as abusive

That’s interesting…. I mean especially for some industries like hotels for examples online reviews are literally what drives consumers…so I get it how some companies can get desperate for a positive review..!! Here is an interesting article I found on the subject: http://growingsocialmedia.com/improve-on line-customer-reviews/

Posted by JacT | Report as abusive