How to ethically improve your customer reviews

By Felix Salmon
May 23, 2011
David Segal's Haggler column this week concentrated on companies which use Mechanical Turk to plant fake reviews -- both positive and negative -- on Yelp.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

David Segal’s Haggler column this week concentrated on companies which use Mechanical Turk to plant fake reviews — both positive and negative — on Yelp. “This is very sneaky,” he writes, “and it’s a continuing problem for Yelp, which is locked in a “Spy vs. Spy”-style contest with fake reviewers.”

But Panos Ipeirotis has found a much more interesting way of using Mechanical Turk to game Yelp; it may or may not be sneaky, but it does seem to give companies a significant advantage — enough that Zappos seems to have spent roughly half a million dollars on it. It’s worth noting that both Zappos and Mechanical Turk are owned by Amazon, so if Yelp has an issue with this, it’s going to be fighting a very large opponent.

The insight here is that the kind of companies Segal is writing about are looking at Yelp in a pretty naive way: good reviews are good, bad reviews are bad. But in fact it’s more subtle than that. In a recent paper, Ipeirotis looked at a long list of variables in hotel reviews, to see which ones were good predictors of customers actually booking those hotels. The statistically-significant variables were these:


Top of the list — the thing you really want if you want your hotel to get booked — is “beach.” But hotels can’t do much about their proximity to the beach. What they can do is address the second-most important variable: readability. Just having well-written reviews, it turns out, is much more important than having good reviews: the rating given in the review was much less significant, as were aspects of the review relating to cleanliness, check-in, service, and the like. And the Value rating on reviews actually had a significantly negative correlation: the better value reviewers said a hotel was, the less likely it was to be booked.

This holds true for other products, too, as Ipeirotis found in another paper:

Demand for a hotel increases if the reviews on TripAdvisor and Travelocity are well-written, without spelling errors; this holds no matter if the review is positive or negative. In our TKDE paper “Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics”, we observed similar trends for products sold and reviewed on

So Zappo’s, instead of getting people to write good reviews, just got them to fix reviews which already existed — deal with spelling errors, correct grammar, that kind of thing. And anecdotally (Ipeirotis got to talking to someone over drinks at WWW India), Zappo’s saw a “substantial” improvement as a result of its investment in cleaning up such things.

Is it fair for Zappo’s to give itself an advantage by doing its reviewers a favor and making them seem more literate than they actually are? I’m not sure. But I’m thinking that I should start correcting spelling and grammar mistakes in my comments section. It’ll only serve to improve what people think of it — and of my blog more generally. Even if those people are very rude about me.

(H/T Hanson)


Comments are closed.