The humbling of Robert Parker

By Felix Salmon
October 2, 2009
Dr Vino's missive from the latest Executive Wine Seminar, which featured not only 15 spectacular 2005 Bordeaux wines, but also Robert Parker, tasting them blind.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

Spending a couple of high-intensity days in Washington, as I’ve just done, is enough to send anybody dreaming of booze. And so it’s quite lovely to read Dr Vino’s missive from the latest Executive Wine Seminar, which featured not only 15 spectacular 2005 Bordeaux wines, but also Robert Parker, tasting them blind.

Parker has rated all these wines, of course. But would his blind ranking bear much if any relation to his official ranking? And how many of the wines could he successfully identify? The answers, I wasn’t at all surprised to hear, were “no”, and “zero”.

To take just one example, Parker identified wine #8, the mainly-Merlot L’Eglise Clinet from Pomerol, as being the mainly-Cabernet Cos d’Estournel from Saint-Estèphe.

Writes Dr Vino, charitably:

A final issue is about points and the nature of blind tasting, a capricious undertaking if there ever were one. Although Parker did not rate the wines yesterday, his top wine of the evening (Le Gay) was the lowest rated in the lineup from his most recent published reviews… For all the precision that a point score implies, it is not dynamic, changing with the wines as they change in the bottle nor does it capture performance from one tasting to the next.

So should we do away with blind tasting altogether? Tom Matthews, the executive editor of Wine Spectator, wrote this on my blog:

Blind tasting is not easy, but that does not mean it’s not useful. With all due respect to Bob Millman, it’s not that tasting blind is “judging from ignorance”; it’s that ignorant judges do poorly in blind tastings.

Millman is the person who runs the Executive Wine Seminar tastings, and he’s much less constructive on the subject of blind tastings than Matthews, and it’s not hard to see why: after all, it does rather seem as though Matthews is saying that Parker is an ignorant judge.

Parker isn’t an ignorant judge, of course. And I daresay he actually did well in this tasting, in terms of judging the wines purely as he was drinking them. He just did badly in terms of identifying what they were, or giving them this time around the same ranking that they got the last time he ranked them. Wine is not a fungible commodity, where one bottle is always the same as the next — quite the opposite. But the fact that wine changes, from bottle to bottle and from month to month, rather defeats the purpose of magazines such as Wine Spectator.

I asked Matthews what he considered a “good judge” to be, and whether there were any downsides to tasting blind. He wrote back:

On judges: Yes, in my opinion, judgment is a quality that lies along a spectrum. “Ignorant” judges lack the context to make useful distinctions; “good” judges have enough experience and understanding to apply appropriate criteria to the wine in the glass. All judges begin in ignorance. If they work hard — taste widely, concentrate intently, read and travel and interview the experts — they may become good. At Wine Spectator, it’s less a matter of “choosing” judges than training them. Before an editor qualifies as a wine critic, he or she undergoes a long and intensive period of apprenticeship with us. The “empirical data” we look for is their consistency of judgment, breadth of understanding and sensitivity to the qualities of the wines they are tasting (faults, structure, flavor descriptors, etc.) Once they do qualify, they are given responsibilty for specific wine types, and as they prove their consistency and expertise, they take on larger tasting beats.

On “downsides”: The challenge in evaluating wine is eliminating externalities that can bias judgment (such as price and reputation) without eliminating so much context that judgment is impossible (since good wine is supposed to reflect its vintage, terroir, etc). Carefully-calibrated single-blind tasting is the methodology that long experience has convinced us is the best and fairest approach. But enjoying wine is a different matter, and in that situation, blind tasting can rob us of information that can supply both pleasure and edification. At dinner parties, I like to serve a wine blind, to get an “unbiased” reaction, then unveil it, so we can learn both from our reactions, and from the wine.

In a world where Robert Parker fails to perform on the criterion of “consistency of judgment”, even in the context of a “carefully-calibrated single-blind tasting”, this does ring a little hollow. Parker didn’t invent the guess-this-wine game that many tastings become, but he is partly responsible for the idea that people who are good at that game are necessarily the people best qualified to judge wine. It’s definitely fun to see him hoist on his own petard once in a while.

More From Felix Salmon
Post Felix
The Piketty pessimist
The most expensive lottery ticket in the world
The problems of HFT, Joe Stiglitz edition
Private equity math, Nuveen edition
Five explanations for Greece’s bond yield
Comments
23 comments so far

I belong to the Duke Ellington school of wine drinking. If it tastes good, it is good.

“So should we do away with blind tasting altogether?”

Wouldn’t the more appropriate question be: “So should we do away with wine rankings altogether?”

I would think the blind tasting would be more valuable than the seeing variety.

This gave me a great big smile. Mr. Parker, how many fortunes of vineyards have been determined by you?

If you really have been BSing all this time then bravo, Mr. Parker!

Posted by Dan | Report as abusive

Is there actually anyone who can reliably identify wines by tasting? Or is that all a myth popularized by the media (examples below), which loves to depict the taster announcing after one sip, “Ah, yes, the 1923 Chateau d’Urbervilles Merlot Gran Prie Fixe, from the shady side of the mountain. A little past its prime, and the sediments of this bottle were disturbed slightly in 1967, but still quite acceptable.”

Examples of this sort of thing that I remember: Donald Pleasance in a “Columbo” episode; Vincent Price and Peter Lorre, in the hilarious “Black Cat” segment of “Tales of Terror”; the Lord Peter Wimsey story “A Matter of Taste.”

Posted by Ken | Report as abusive

Count me as unsurpised. I have long known that Parker regularly purchased wines from careless retailers that let wine get hot. Sycophants always said he can taste for that, but I always doubted it.

Posted by Anonymous2 | Report as abusive

If you meet someone who thinks that he/she knows all the answers, or is the sole arbiter of any subject
RUN

Posted by historygypsy | Report as abusive

Ken: In one of his comedies Louis de Funes playing a restaurant critic managed to ID a wine the way you describe it just by looking at it. (He had to do this as his taste was sabotaged by the evil competition.)

Here is the scene
http://www.youtube.com/watch?v=bUDeD57wq Ms

Posted by IF | Report as abusive

I’m disappointed. I’ll never read another Spencer novel.

Posted by dSmith | Report as abusive

Over time, we have seen that Parker is just not good at blind tasting. That does not make him a bad critic, but he should stop the claim that the Wine Advocate tastes wine blindly, because, asit has been shown each time he does an EWS event, he is just not good at blind tasting.

I’d be curious to see the “empirical data” that Tom Matthews thinks prove the “consistency of judgment” of Wine Spectator’s critics. I suspect the empirical data would show otherwise. One example I remember being discussed was 1997 California cabernets; all of the top wines as rated by WS’s James Laube on release scored substantially lower in his retrospective tasting a decade later, and his top wine a decade later had not been rated especially highly in his original tasting.

@Daniel Posner: Parker does not claim that all of his tastings are done blind, and says this specifically in print and online. You know this. Obviously, it’s impossible to do so when he is tasting at wineries sur place, which he often does. If you want to trash the guy, at least get your facts straight. I would add that anyone who has blind-tasted a bunch of young Bordeaux of cru classe Bordeaux from a great vintage knows that trying to nail particular chateau is a fool’s errand. You have wines made with same techniques, from the same grapes, from great vineyards that may be yards apart. It would be a surprise if were NOT almost impossible to do. Guessing the wine may not be meaningful, but it’s fun to do, which is why Parker does it. Frankly, I commend him for that, knowing that he will have to take flak from people who should know better. I’m willing to excuse Felix, who at least gets things right in his areas of expertise, which unfortunately, do not as yet include wine.

Posted by maynardGkeynes | Report as abusive

I recently tasted a Molly Dooker that I didn’t care for. Funny thing is that I loved this wine in June when I knew what I was drinking. I guess Parker and I are in the same boat.

Posted by Dan H. | Report as abusive

No general wine critic can really identify the vintage of a wine, that’s pure myth. And the fact that a wine might be a 98 one day and a 97 the next is something Parker is completely open about,though admittedly a lot of readers treat his ratings like a baseball score. Felix is completely misunderstanding the event if he thinks differently.

What I really want to see in these blind tastings is whether he would fail to identify an 89-rated wine against a bunch of 98-rated wines. That’s the thing that would really show these magazines to be worthless.

Posted by WoofWoof | Report as abusive

Felix,

Thank you for quoting my remarks accurately and at such length.

I have read Dr. Vino’s report of the Bordeaux tasting, and while it must have been humbling for Mr. Parker not to have guessed a single chateau correctly, this (alas not uncommon) outcome does not in my mind invalidate blind tasting.

WoofWoof’s comment is apt. All the wines in this Bordeaux tasting were originally rated at 95 points or higher. That fact was known in this tasting. I would be more interested in the results of a second blind tasting if the original scores were not known, and were more heterogeneous. Would all the ratings again fall into the same score bands? And would the tasting notes be consistent? That to me would suggest a “good judge.”

Or say the original scores were given non-blind, while the subsequent tasting was blind. A “good judge” would give consistent ratings and tasting notes. Any significant inconsistencies between the two might be attributed to the externalities blind tasting eliminates (and might cast doubt on the quality of the judge).

Since neither case applied, the tasting Dr. Vino describes does not seem to be a good test of the validity of blind tasting. I will say, however, that if the tasting notes Dr. Vino supplied in his report were written before the wines were disclosed, then based simply on his descriptions, I would consider him a “good judge” of young Bordeaux.

Thomas Matthews
Executive editor
Wine Spectator

Posted by Thomas Matthews | Report as abusive

What wine needs is a little pill you take before drinking that conjures in your mind the ambience and mood which imparts a significant part of taste. Not a pill that turns dreck into gold but one an enhancer that mirrors the findings of behavioral econ.

Posted by jonathan | Report as abusive

This is not the first time that a blind tasting has gone “FAIL” – and it won’t be the last. SUch is human nature. What differs here is that the main failing is the proverbial infallible one, who has in the past claimed a near photographic palate memory of wines he has tasted, particulalry within his area of expertise (BDX).

The reverberations of this event will, I think, be different within and beyond wine circles. Within, most people who actually understand how wine criticism works will not be especially shocked; and they will continue to follow whatever system they do.

Outside wine-geeky circles, this will serve as a wake-up call to the fact that numbers are not only hopelessly imprecise and irreplicable. Robert Parker’s “FAIL” will help draw attention to eh failure of numbers in general, as well as the hideous practice of retailers and marketers plucking numbers and flaunting them as if they are in any way definitive. In nearly every instance that a wine rating is foisted upon the world, the number itself represent no more or less than one man’s preference based on one set of wines on one day — and arrived at without a crumb of food.

Wine criticism, involving not only ranking but also description and guidance for usage in context, will always be a welcome aspect of the Wine Life. But the proliferation and abuse of singular numbers is absurd. Let’s hope that the future of wine criticism and marketing alike come back to earth from the senseless 90-point stratosphere.

Two points:

The idea that there are 100 different discernible linear gradations of wine quality is ludicrous at best. If you were trying for repeatability of ratings, especially between experts, you *might* manage 10.

The point of ratings is not to provide factual accuracy, but to give some pleasure to the reader, which is why he or she buys the magazine. That the ratings aren’t accurate (or more likely, can’t be accurate) is irrelevant if the audience is pleased by seeing the numbers. If finding, buying and consuming that 98 or 99 rated wine pleases the audience, why is it important if all the ratings above 90 are more or less random.

After all, is it a crime that wrestling isn’t real? Is the audience being ripped off?

In both cases, the audience is buying *for* the fantasy. If they’re happy, then it’s good for all.

Posted by Tom West | Report as abusive

The only thing Parker has actually taught us is that he’s especially cavalier with his reputation. Any other critic, no matter how experienced, has as much potential to perform as poorly as Parker did at this particular tasting. They’re all just smart enough not to put themselves in such a situation.

This is also why normal blind tastings for magazines or websites are designed so that the taster doesn’t score erratically (re-tastings occur, wines of a certain region/variety/vintage are always grouped together). In other words, the very system itself is designed to protect the critic’s reputation as much as–or even more than–serve the consumer.

As much fun as it might be to attack Parker in this instance, consumers should know better than to bestow excessive trust and credibility upon wine critics. They’ll only wind up disappointed.

Is there any empirical data on how much wines vary from bottle to bottle? I know I’ve had some wines that were so-so the first time I had them, and great the second, or vice-versa (same label, same vintage, same store). I’ve always wondered whether that was a result of variability in how I tasted the wine, or variability in the bottles.

Posted by Mungo Bola | Report as abusive

My wine expertise is limited to knowing the best vintages of Charles Shaw and Boone’s Farm. I can however tell you with metaphysical certainty which Anheuser-Busch location crafted the bottle of Budweiser I am drinking before it is half way gone. I am not as precise with cans and my ability with kegs has decrease exponentially since my youth. Friendly tip: Avoid the Houston beer brewed during the summer and stock up on the Syracuse beer crafted in the fall. The Ft. Collins beer is good year round.

Posted by Carlos Brito | Report as abusive

My favorite quote on blind tasting comes from Importer Kermit Lynch. “Blind Tasting is to wine drinking what strip poker is to love.” In other words, it is a fun game, but totally misses the point.

http://www.austinbeeman.com

Posted by austinbeeman | Report as abusive

Parker creates a financial wave, just like any high-octane equity analyst. You can either choose to ride it, as investors do (including me), or you can ignore it.

If you happen to like the wines he likes, he can also highlight some good things that remain at drinking, rather than investing, levels – I have done this too. Whether he can taste blind, or ride a unicycle, is immaterial.

Posted by TheToothpick | Report as abusive

It was an eye opener to see how hard it was to tell cab, merlot and carmenere from each other at a blind tasting which included some people in the trade.

I always question numeric ratings because the point scores and price almost always go hand in hand. My $13 zin was three times more popular than a $30 zin at a recent blind tasting.

Posted by RaScott | Report as abusive
Post Your Comment

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/