Tasting wine blind

By Felix Salmon
September 12, 2009
Executive Wine Seminars, on Wednesday, and I knew I had to meet him, so I did just that, this evening. Bob's been running blind wine tastings for decades now, and so he knows just what they're good for and also what they're really bad at. Here's a bit of what he wrote to me:

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

I got a wonderful email from Bob Millman, of Executive Wine Seminars, on Wednesday, and I knew I had to meet him, so I did just that, this evening*. Bob’s been running blind wine tastings for decades now, and so he knows just what they’re good for and also what they’re really bad at. Here’s a bit of what he wrote to me:

It should be obvious to any thinking person that blind tastings necessarily favor–on a group vote basis–wines which offer immediate pleasure and gratification. Left to their undirected devices, the senses will almost always gravitate to the obvious and miss the subtle. I have fallen victim to the sweeter-is-better trap several times myself. At blind tastings that include left bank and right bank young Bordeaux, the right bank wines almost always garner higher scores from the crowd. Young Merlot simply tastes “better” than young Cabernet Sauvignon. Softer, sweeter, more flattering.

When I did a blind tasting of Pinot Noirs a couple of years ago, I got really excited about the eventual winner, the 2005 Heron. I ended up buying quite a lot of it, and sometimes ordering it in restaurants or bars as well, and, weirdly for a wine which everybody thought was spectacularly good, it didn’t grow on me at all — quite the opposite.

Part of that is maybe just that most wine deteriorates with age, and that the ’05 was better when we first tasted it than when I drank it a year or more later. But another part of it, I think, is that the kind of wines one loves in blind tastings are not necessarily the kind of wines one actually likes to drink in real life. As Bob says, they tend to the soft, and fruity, and sweet. If you normally like that sort of thing, then great, but if you tend to prefer something a bit more austere or elegant, then you might well end up doing yourself no favors at all if you taste a lot of wines blind.

This evening, Bob ordered a gorgeous terroir-based Beaujolais, after which we moved on to a big Californian old-vine Zinfandel. We both preferred the old-world wine, but as Bob said, if you put the two next to each other in a blind tasting, there’s no doubt which would win. In contemporary art terms, it would be a bit like exhibiting Fred Sandback next to Jeff Koons. Even when you’re tasting like against like — for instance, when you’re tasting young Bordeaux — you’re just as likely to steer yourself wrong by tasting blind as you are to find a hidden gem, since the best Bordeaux wines just don’t do very much when they’re young.

“The problem with blind tasting is that you’re working from a position of ignorance,” said Bob. If you know exactly what it is that you’re tasting — a young first-growth wine, for example — then you can taste it in that light. Similarly, if you know that you’re looking at an Ad Reinhardt painting, you’ll be willing to spend a few minutes with it so that you can appreciate its subtleties. If you didn’t know it was a Reinhardt, then you’d probably just read it as a black monochrome and move on. Or think about someone like Joseph Beuys: the whole point of the art is that it’s multi-layered, and responds slowly to the viewer, who has to think things through.

Much wine — and most great wine — is similar: it grows on you, slowly. But in the artificial environment of a blind tasting, where you’re running through a dozen or more wines, it’s impossible for the vast majority of us to do such wines justice. Hell, its hard enough to do even one bottle of great wine justice: while every so often, while you’re drinking a bottle, it all comes together spectacularly, there will also always be times when you take a sip absentmindedly between bites and miss a lot of the beauty and flavor.

On top of that, blind tastings by their nature become guessing games, and the people doing the tasting, rather than simply approaching each wine with an open mind, are constantly asking themselves which wine it might be, whether it’s famous, or expensive, or maybe even the one that they brought to the tasting. Such thoughts are not conducive to the appreciation of great wine, just as someone looking to see whether a painting was genuine or not is likely to miss out, while doing so, on much of its aesthetic appeal.

In any case, the various different factors which go into the enjoyment of a wine are so multitudinous that when you try to eradicate them all in order to allow different wines to compete on a level playing field, you at the same time eradicate much of what makes a wine so enjoyable in the first place. You might love your spouse’s [insert body part here], but it would be pointless and invidious for someone to test that love by presenting you with a series of carefully anonymized body parts and asking you which one you liked the most.

On my first anniversary, my wife and I drank a spectacular bottle of wine which was given to us as a wedding present by a good friend; what’s more, it came from the cellar of a wonderful restaurant and we drank it at that selfsame restaurant, with delicious food and friendly service and in general everything going right. We’ll both remember that bottle for years, it was everything you could ever hope for from a wine, and more. That’s by far the best way to drink great wine: with good food, on a special occasion, with people you love, purely for enjoyment. If you take most of that away, and drink wine blind, surrounded by serious men spitting into buckets, you’re doing something qualitatively very different indeed. And it should come as no surprise that there might not be much if any correlation between how much you like a wine in the former context and how much you like it in the latter.

There’s also a pretty strong argument to be made that blind tastings are positively bad, harmful things: that the hot, sweet, oaked, fruit-forward, soft-tannined wines which tend to excel in blind tastings are precisely the wines that everybody in the world is trying to make right now. Blind tastings, in many ways, might well have caused the homogenization of global wine culture — something which pretty much all wine lovers abhor but which seems at the same time to be unstoppable. And they’ve done it through the mechanism of the 100-point tasting scale, as invented by Robert Parker.

Wines sell based on how many points they garner, and they garner points by being tasted, blind, by critics like Parker. It used to be that if one importer or wine merchant didn’t like your wine, another one would. Nowadays, it doesn’t matter what those individuals like personally: it matters what the blind tasters at the Wine Advocate and the Wine Spectator like. And the importers and merchants flock to the high-point wines, because those are the wines that their customers want.

What is blind tasting good for? Well, for one thing it’s very good at showing how important knowledge of price, as opposed to price itself, is as a contributing factor to a wine’s perceived quality. If you know that a wine you’re drinking is expensive, you’ll probably like it much more. If you’re deceived into thinking that a wine is expensive (if someone poured Yellowtail into a Lafite bottle, say) you’ll like that much more, too. And if someone poured Lafite into a colorful screw-top bottle, you’d like it less.

When I say, then, that in wine there’s no correlation between price and quality, what I mean is that there’s no correlation between price and quality except for in the 99% of cases where in fact the correlation is very strong — the cases when you know, more or less, how expensive the wine you’re drinking is.

I’m trying to train myself out of that ingrained mindset, by drinking quite a lot of cheap wine and buying large quantities of the good stuff. And there really is a lot of good cheap wine out there. But I know that I do still have the same prejudices as everybody else, no matter how much I write about negative price-quality correlations. If I open a cheap bottle and I don’t think much of it at first, I’ll assume it’s not very good. On the other hand, if I open an expensive bottle and I don’t think much of it at first, I’ll let it breathe, I’ll revisit it later, I’ll try to see if I can discern some subtlety and sophistication which might not have been immediately apparent. And if I look hard enough, I’ll probably find it.

*Um, I mean yesterday evening. I wrote this on Friday, but finished it late, and for some weird reason my internet seems to go down every evening at about 10pm — it’s done it four nights in a row. I spent an hour on the phone to Time Warner tech support on Thursday night, to no avail, and they’re sending someone out on Tuesday morning, when the internet will probably be working fine. Any ideas what the problem might be? The internet doesn’t go down entirely — the lights on the cable modem are still right, and I can occasionally download emails or the Google home page (which takes literally minutes to load). But it’s unusable, and the signal, as seen by Time Warner’s techs, is all messed up.

Update: Thomas Matthews, the executive editor of Wine Spectator, leaves a comment defending blind tasting and saying that if you want to be objective about things, “there is no better methodology”. I agree. But I think that objectivity is overvalued sometimes, and in any case it’s important to be aware of the weaknesses of the blind-tasting methodology, even if you can’t come up with a better alternative.

Daniel Posner notes that Robert Parker himself rarely tastes wines blind any more — this is true, and I think does speak to the limitations of blind tasting.

And londenio draws the comparison between wine and cola: Pepsi generally wins in blind tastings, ‘cos it’s sweeter. But most people still prefer Coke.

16 comments

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/

Some of your criticisms miss the mark.

1) A blind test will tell you which of two (or more wines) you prefer today. It is not expected to be a good predictor of which wine you will prefer years from now

2) If you are not careful in blind tests, taking time, giving the wine a chance to breathe, etc., you will not get good results. The fault is not blind testing, it’s sloppy testing.

3) If high price (or label or ambiance or whatever) affect our perceptions, does that mean the wine is really better? This is not an easy question. Consider the issues associated with placebos in drug testing. Patients who take placebos often experience improvement, sometimes dramatic improvement.

Posted by foosion | Report as abusive

I agree with foosion. It’s a little silly to depict a blind taste test as somehow corrupting your choice. Your anniversary experience really cinches it.

For you, the idea (its story) of a great wine is as important as how it tastes. There’s nothing wrong with that, part of what you’re consuming is that story, and it brings you satisfaction, which is all that matters.

I read an economist who explained that’s how we should look at people who play the lottery. To many of us, it seems mind boggling that they would play a game with such horrible odds. But they don’t see it as an investment. They are consumers. They are buying a chance.

We all spend money on things which aren’t logically defensible. Just accept it.

Posted by Bob_in_MA | Report as abusive

My sister has Time Warner. It was out. She says she called tech support and first thing she heard was that they had systemwide problems. I believe they had a big dos attack or whatever they call it a month or two ago which some bright british chapsticks traced it to the chinese government. So faulty routers, hackers, sunspots, or tactical neutron bombs. They’ll never tell you for sure :)

Mr. Salmon,

I would not argue that all wines should be served from paper bags. There are many occasions where blind tasting is not appropriate. However, when the goal is objective assessment of character and quality, there is no better methodology.

Blind tasting is not easy, but that does not mean it’s not useful. With all due respect to Bob Millman, it’s not that tasting blind is “judging from ignorance”; it’s that ignorant judges do poorly in blind tastings.

Good judges know that the “immediate gratifications” of, say, residual sugar and high alcohol are not necessarily indicators of underlying quality. Experienced critics know how to assess wines against their peer groups and give useful judgments about their character, their typicity and their future.

However, even good judges can be influenced by (more or less conscious) biases in a non-blind setting: we give the famous artist, or the expensive wine, the benefit of the doubt, while discounting the virtues of an unknown artist or an inexpensive wine. The studies documenting these biases are simply too numerous to ignore.

At Wine Spectator, we believe that a trained judge will reach the best assessments from blind tastings. Blind tasting is our guarantee to our readers that our judgments are not influenced by price or pedigree, that our reviews are free from any bias or conflict of interest. Every wine gets an equal chance to show its best and be evaluated on its own merits. Blind tasting is the methodology that best guarantees honesty and fair play.

Thomas Matthews
Executive editor
Wine Spectator

Posted by Thomas Matthews | Report as abusive

Felix

You ought to edit your statements, because if you have done any reading as of late, you will see that Parker seldomly tastes any wines blind.

The point is that any critical methodology falls apart if you use it to measure things it was never designed to measure. Blind wine tastings are useful in a variety of situations–allowing people to compare and understand terroir differences (compare Zins at comparable price points from different growing regions), or helping people make better price/quality tradeoffs (compare different price points of the same varietal blind, before deciding whether the quality of the higher priced wine is worth the difference). It was always absurd to think that blind tastings could be used to compare dissimilar varietals, or wines with different aging characteristics. 100 point ratings have some legit value within narrow comparison groups; the problem is not just extending them to absurdly wide groups (“the 90 point reds of 2008!”) but the failure to acknowledge or explain the underlying standards, where the real villian isn’t Parker but his many imitators

Posted by Hubert Horan | Report as abusive

If you were a wine, which wine would you be?

I read this entire post thinking I was reading Jamie Goode, whose blog I also read in Google Reader. I thought the comment about drinking lots of cheap wine was a bit strange for a critic, but I thought maybe he’s going through a phase. Then I got to that last bit about Time Warner, and I couldn’t decide if Time Warner was now providing access in England or if Jamie’s moved to New York. So I scrolled to the top and lo and behold, it’s Felix. Hmmmmm, I guess I should read that again, right? Thinking the post was written by one person obviously affected how I read it. Then I realize: it’s the same thing with wine! I just had Yellowtail (Felix) poured into a bottle of Lafite (Jamie Goode). And it messed with my palate.

Just kidding, Felix. But I thought the mix-up was oddly topical.

Posted by Jon | Report as abusive

This is a testable hypothesis, maybe a good graduate level thesis. Hold a (regular) blind taste test, then distribute unlabeled cases of wine and check back in a year.

Posted by ang | Report as abusive

I find the notion that a properly conducted blind test might lead me to prefer a wine which I actually do not like — well, to tell the truth — confusing. I suppose I do not understand what you mean by ‘like’ and I must wonder (in a friendly and non-confrontational sort of way) whether you do. (But don’t feel bad: most people appear confused about the meaning of that word). cheers

Posted by gawain | Report as abusive

In blind tasting, most people prefer Pepsi. When they purchase, most people prefer Coke. Branding experts like to claim that all the investment Coca Cola Company has made in the brand explains this reversal. However, another explanation is that Pepsi tastes sweeter and therefore we like it more when we drink small amounts.

This excellent post reminds me that the same happens with wine.

Posted by londenio | Report as abusive

I agree with you wholeheartedly. I’m a tea-blogger, and one of the things I used to do more of, but less so nowadays, is tasting teas blind. In the case of tea the problem is even more acute, as there is much less collective experience with drinking tea, with more variables involved (unlike wine, which has a bottle to take care of most elements, tea does not). The issue, ultimately, is one of changing standards. As one of the comments above pointed out, it is useful to rate wines (or teas) that are similar on a scale of 100 and make that a fair comparison, but it is much more difficult to do the same once you cross regions and styles.

Interesting reading indeed. Blind tasting is a very humbling experience for all involved . . . invariably someone will turn out ‘liking’ something they thought they never would . . . inevitable.

I agree that there is a time and a place for blind tasting – it is an ‘educational’ tool but not always the most ‘fun’ thing to do, especially at something like an Anniversary dinner!!!

Wine tasting has so much more to do with just the wine itself – it’s the ‘romance’ of the setting, the food served with the wine (if food is to be served). the company you are with, how talkative or quiet the people are that you are tasting with, etc . . .

I can say that if you can remove as many ‘picture clues’ about what you are tasting, it makes the task that much more educational . . . . try dark wine glasses and a darkened room sometime, and see how many wines you pick out as red that are actually white and vice versa!!!!

Thanks again . . .

Cheers!

londenio

you point out well the difference between liking and preference; we often prefer things we don’t actually like (such as a better paying job, for example; or pleasing mommy rather than self); i suppose this is the case here: one might prefer (for whatever reason, fill in your own blanks) to drink an expensive mouth-challenging wine, but actually not like it; a blind test will rudely reveal the pretense; it doesn’t mean the blind test is wrong; au contrare.

(think of all the jazz-loving people sleeping through their subscriptions at the opera at the met. they prefer to go to the opera, yes. it don’t mean they actually like it).

Posted by gawain | Report as abusive

You’re conflating multiple things, of course, and European snobbishness, to be perfectly blunt, is just one of them (if a California wine would beat a French one in blind tasting, that must be an indictment of blind tasting rather than an endorsement of California wine).

“Sip tests” are well-known to be less than ideal for evaluating products that are consumed in larger quantities than a shot glass–the canonical story is of the Pepsi Challenge and the New Coke debacle that ensued. The blindness doesn’t enter into it–it’s the quantity in the test that doesn’t work well. If the way to appreciate wine is by the glassful, over the course of an evening, then that is how tests should be conducted…and if you can only try two or three wines in an evening, rather than twelve, well, there you are.

To say that tests shouldn’t be conducted blind is to say that tasters should be allowed to overrule the evidence of their senses based on a corpus of past experiences, prejudices and conventional wisdom. The problems with this approach should be obvious after a moment’s consideration. You might do nearly as well by simply displaying the bottles to the judges unopened, and letting them write down their scores based on how the wines “should” be rated. It’s an interesting idea, and of course it would spare the egos of the tasters from such embarrassments as the Judgment of Paris, in which California wines were praised as the embodiment of centuries of French genius, and French ones criticized as jumped up Colonial pretentions. No wonder blind tasting is frowned upon.

Posted by Craig | Report as abusive

Felix-

Good sentiment, but you are confusing two issues: whether you know thw price of the wine; and whether you taste half a glass or half a bottle. The second matters more than the first.

-JAFDC

Posted by JAFDC | Report as abusive