Reviewer's Review: Whisky Advocate

What is feckin whiskey doing on the net's leading independent rum website? There's a reason, read on, but it's not my fault! Honest...
Post Reply
User avatar
Capn Jimbo
Rum Evangelisti and Compleat Idiot
Posts: 3550
Joined: Mon Dec 11, 2006 3:53 pm
Location: Paradise: Fort Lauderdale of course...
Contact:

Reviewer's Review: Whisky Advocate

Post by Capn Jimbo »

An Analysis of the Whisky Advocate (and blog)...


A quickie! And don't you love quickies? Sure ya do. Normally I wouldn't waste my time on reviewing a whisky site (hint: this is a rum site), but one of our valued members posted a mild defense of Canadian Whiskey on a thread names "Canadian Whiskey: The Spam of Spirits".

Now he didn't counter the premise: that Canadian Rye Whiskey is generally a blend based on thin grain spirits, often doesn't contain rye, is bulked up with a bit of pot stilled or other whiskies, and up to 10% of damn near any additives you can think of. However tasty, it's a mixed drink in a bottle, lol.

OTOH he did encourage me to give Canadian Whiskey yet another try, even suggested two based on a couple of brief but very reviews from a website called the "Whiskey Advocate". And blog. Upon reading these little tidbits, the hairs on my neck immediately arose.
Why?

Simple. As a former copy writer/marketing manager I tend to sense shilling within a descriptor or three. And if it reviews like a duck... but to be fair, let's do an analysis and see! First let's look at the Advocates' rating scheme:
95-100
A classic! All components are balanced appropriately, with the complexity and character expected in a classic.

90-94
Outstanding! One of the best for its style. Distinctive.

80-89
Good to very good. Plenty of character and no identifiable flaws. Worth seeking out.

70-79
Average. No unique qualities. Flaws possible.

60-69
Below average. Major flaws. Avoid.
And the envelope please...

. . . . . Image


Wow! Talk about a high score bias! Almost all the scores are well above average, with way, way too many spirits earning scores in the coveted 90-100 range.

Let's go on...

This is so far from a normal distribution's bell curve it isn't remotely funny. In a normal distribution the lowest and highest ranges should be roughly equal. Here they are 2 vs 401 whiskies. Now that's distributive bias. And the middle two ranges should be likewise, here 97 vs 758 whiskies. Actually, the largest range here should have been 70-79 ("average" whiskies), not 80-89.

The imbalance should be clear.

You'll also note these:

1. The "reviewers" (read copywriters) have scored 1258 whiskies. That's a ton. It is claimed that many other whiskies were sampled but not reported.

2. Each "review" conveniently publishes a link to obtain - are you sitting down? - to print out a "shelf talker". I'm serious. Can it get any clearer? No way!

3. The website states "Every attempt will be made to procure products that are in good condition and unadulterated. To insure this, distillers, importers, and wholesalers are encouraged to submit samples for review". In other words, freebies?

4. The site states that products are rated in comparison of whiskies of the same style. Thus a blended Canadian whiskey can obtain the same 95 as a 40 year old Islay. To me, this seems yet another sop to the trade.

So far, not so good.


But is there a counter to all this?

Maybe. First of all, the site claims some really impressive reviewers:
Due to the increasing amount of new whisky releases worldwide, Whisky Advocate has expanded its number of Buying Guide reviewers to five people: Dave Broom, Dominic Roskrow, Lew Bryson, Gavin Smith, and John Hansell. Everyone on this list is an authority in the whisky world, an experienced whisky reviewer, published author, and veteran writer for Whisky Advocate, formerly Malt Advocate.
(Emphasis added)

The mere fact that Dave Broom appears on this list gave me pause. I've followed his reviews for years - like Michael Jackson, he's calls 'em as he tastes 'em. His distributions are absolutely normal (see Reviewers' Reviews in the Scuttlebutt section). So why aren't the Advocate's scores more balanced?

Can it be because the site also claims "The tasting notes and ratings published are those of John Hansell, publisher and editor of Whisky Advocate magazine". Now we're gettin somewhere, read on...

The Advocate also makes a strange claim:
Please note that we sample far too many products to publish reviews on all of them. The products listed in our Buying Guide usually reflect our most positive experiences. However, we will also review products we don't particularly like if there is a point we are trying to make.
Aha! Let's deconstruct this. The Advocate, no doubt keenly aware of what appears to be a high scoring bias, may have anticipated posts like this one, and wants us to believe that there are so many, many, many, in fact "...too many products sampled to publish reviews on all of them" - so - they just publish the positive experiences.

Are you buying that? Why not publish all of them, or at least the most available ones, and take up another few more measly megs on your provider's hard drive? Doesn't cost another provider's pretty penny, fairly easy to do, and - uh- just might be valuable to whisky drinkers. Really now. Hell, they did the sampling, that's the hard part! Sorry.

And perhaps realizing that someone like me also just might make that point, have they then reversed course to assure us that "...we will also review products we don't particularly like if there is a point to make"?! So out of 1258 reviews (and apparently many more simply not published), just how many crappy whiskies did they select and report "to make a point"??

Two. Yup, just fackin two! Is this reasonable? Or does this amount to just shilling style bullshit? You've been given both sides, you decide...


*******
Special Note: The Advocate was contacted by a member, and responded in defense. Page down about 4 replies to "Late Breaking News! The Advocate's Reply"! You won't believe it!
Last edited by Capn Jimbo on Wed Dec 07, 2011 12:49 pm, edited 7 times in total.
User avatar
Uisge
Cap'n
Posts: 178
Joined: Sat Nov 05, 2011 4:32 am
Location: Marvelous Madera Ranchos, CA

A short defense of John Hansell

Post by Uisge »

If there are whiskies that have been sampled, but not review due to falling below the threshold for what is deemed "acceptable" or "decent", I don't have a problem with that. I don't need to know about a Yugo if I'm looking for a Corvette (or put in your preferred vehicle of choice), if you get my drift.

I DO want to know about good, quality whiskies, which is what John does,and so does Dave Broom and others at Whisky Magazine, which I view as the European perspective on whiskies and things surrounding it ( like travel, foods, "lifestyle" things...hey, that's "shillery", too!).

I consider John to be the American whisky reviewer equivalent of the late Michael Jackson (and I've personally met Mr. Jackson twice before he passed), both for the amount of time he has had in reviewing whiskies, and for what he got his start in reviewing, which is beer (MJ started reviewing at the start of his career, for a newspaper if I'm not mistaken).

John himself does not equate himself with MJ or Broom, having written a post almost a year ago with this in the opening paragraphs:
"Well, to be honest, I know that I’m not a great whisky writer. I read stuff by Michael Jackson (may he rest in peace), Jim Murray, Dave Broom, and more like them in Malt Advocate magazine and they just amaze me. They, are great whisky writers.

I also know that I can’t describe whiskies as eloquently and colorfully as the best ones. (I refer you to the names above once again. They are all brilliant at what they do.)"
Full post found here.

Is John Hansell a shill? Well, check out this page titled "About John" here. He most certainly is a shill for what was Malt Advocate, is now Whisky Advocate and the Whisky Live tasting events, but that is understandable as those are his "babies", which he started. He does get paid by revenues generated by sales of or to the above listed.

John was successful enough with what he produced that Shanken Communications (Wine Spectator and Cigar Aficionado) bought Malt Advocate magazine to join the stable with the other two magazines last year.

As for being a shill for the producers, you might want to read a few of his posts (gathered under the banner of "Things which piss me off", written about this time last year) found here, here, here, here, and a summation of all the above here.

John is passionate about whisky (and beer), to a similar level that you are passionate about rum, I would wager. I respect both efforts put out by yourself and John, and I share them to some degree, hence my seeking out your site, Whisky Advocate's blog and magazine, and also why I subscribed to Whisky Magazine for a decade, until earlier this year.

Ultimately, that I value all the above for me to learn and better form an opinion is a form of an opinion, and nothing else; shillery, to use your term (please don't take it as a slight).

As I am not paid for giving my opinions, the things of value I bring are either my earned dollars or time invested in reading and participating at a blog.

Just as I wrote to you about Canadian whiskies, you might reconsider your position on Whisky Advocate and John Hansell. He doesn't need me to defend himself and what he does, as he is doing quite well at living off doing that very thing he loves (I refer you to what I previously pointed out about Malt Advocate being brought into the Shanken Communications stable of magazines), and that does tend to lend a form of credibility.

All the above is an unpaid, non-solicited opinion, so take it in the manner that it is given: Freely.

Thanks for your efforts and site, Cap'n.
User avatar
Capn Jimbo
Rum Evangelisti and Compleat Idiot
Posts: 3550
Joined: Mon Dec 11, 2006 3:53 pm
Location: Paradise: Fort Lauderdale of course...
Contact:

Good response!

Post by Capn Jimbo »

A well considered and detailed response, thanks.


Here's the deal. Regardless of what anybody purports to be, wants to be, or thinks he/she is, the proof is in the dram. Often the reviewer him/herself is not aware of their bias until they see an analysis.

In our post in the Scuttlebutt section, "Normality and Idiocy: Bell Curves 'n Rum Reviewers!", the subject of bell curves and normal distributions are discussed (here). I urge you to read this right now. It begins:
It's just life baby...

"Life is full of natural phenomena: height, shoe size, and people. And some that are but one step removed: SAT test scores, bond investment return, employee performance and rum reviews. What's common to all these is how the results are distributed...
Almost all phenomena, but especially unbiased personal performance falls into a normal distribution. For example, Dave Broom's reviews have been analyzed and are completely normal. In the Reviewer's Reviews in the Scuttlebutt section you will find varying levels of possible bias, from none to lots.

A common laymen's (and reviewer's) response is "...but I choose or report just the better spirits!". No matter, choose just crappy spirits, or just stupendous ones, the result will be the same: a nice normal distribution, aka bell curve. As put in "Normality", it doesn't matter where you aim at the target, your results will scatter about your aiming point, a few dead center, a few way off, and most balanced inbetween.

That's life and reviewing results.


Bias doesn't have to be intentional

A well known reviewer who shall remain nameless was shocked to see his analysis, recognized the bias in his scores (not his intent) and decided to rescore all of his many reviews, a huge project. This brings up a very important point.

Bias is not always conscious. A real whisky lover may tend to overvalue too many, too often out of simple love for the spirit. Sort of an "it's all good" attitude. Reviewers who are dependent on industry support and free samples may pull their punches, a little or a lot. Still...

It all comes out in the analysis.

If there's little bias (and there's always some) it shows, ibid if there's a lot. No reviewer presents a perfect bell curve, but as our article "Normality" makes clear, the more samples, the more an unbiased performance will approach an ideal distribution.


Shill? You decide...

To be fair, I never called Mr. Hansel a shill. What I did say is that as a former copywriter, I tend to recognize shilling, but continued to say that only an analysis would tell the story. The analysis was performed, and it implied possible and significant bias. In Hansel's defense I then did my best to counter the analsyis, however unsuccessful that effort proved to be.

In the end, the only reference to shilling as regards Hansell was to pose the question "Is this reasonable? Or does this amount to just shilling style bullshit? You've been given both sides. You decide...".

I appreciate that you like Hansell for his affiliations and the history you perceive, but I'll leave it with this: a "90" from Hansell is not all that unusual for him, and is thus not nearly as meaningful as it might otherwise be. And though I didn't mention it, his top ratings showed a preference for extremely high priced spirits (single malt whiskies scoring in the 90's averaged about $1400 - each!).


Couple other notes:

I've worked for magazines and other media, in both copywriting and marketing. Success in this world is mostly dependent on ad sales, and I'll tell you this. The sales department trumps the editing every time. Magazines are dependent on ad revenues and sales, and trust me, I've had copy pulled before publication by top management based on leaks to the ad buying subject.

"It's only life, baby!" Same is true of Rumfests and their competitions. Speak negatively of a few distillers and the word gets out, and your table sales go down. I speak from personal experience here as well.


What does Hansell say?

Thanks for the links, but Hansell doesn't say all that much, rather speaking in self-serving generalities. He makes the same claims we all do...
I can't be bought... If a whisky company makes a great whisky or is doing something really cool and innovative, chances are I will write about it. And if they make a product which I think is flawed or they do something questionable or misleading, chances are I will write about that too.
Sounds great, but does he? Out of 1200 plus reviews, he reported only 2 that he'd reject? Is that credible? In his defense, perhaps his articles are more balanced, but the reported reviews don't seem to be. Let's continue...
Still, many producers... get upset with me when I don’t give them a great rating... there are recognized voices in the industry reviewing whiskies whose ratings average over 90. It’s no wonder that they get upset when I only give them an 86 (gasp!) which, in my view, is a very good whisky.
Poor John, facing all that fire for giving an 86! He's not being faulted here for that - but he is being faulted for giving so damn many of them in the 80's- almost 800 - not to mention more than 400 in the 90's! Begorry! Yet only 2 really bad whiskeys, and only 90 or so "average" ones? Does he know what "average" really means? And last...
When I catch someone pumping up their brand, I call them on it. I’m not going to list specific brands. It will take me a long time to go through my 987 posts and your 16,610 comments and find them, and I don’t have the time right now.
This is the best one.

Of course he doesn't have the time, and probably not the interest. Nor do most reviewers, including me. You have no idea how many hours went into each analysis, having to read and count every review and score, then enter data into a spreadsheet, to create the handy dandy summary chart, and to review and discuss other important trends.

But we did find the time - and went through each of his 1200+ reviews and scores. It was the biggest analysis, by far, conducted to date, and more than all the rum reviewers combined. Huge! And the results:

1. He names the good and great ones, nearly 1100 of them, but calls out only 90 or so of the "average" whiskies, and - gulp - only 2 bad ones.

2. Each one has a link to a "shelf talker". That alone is a flag raiser.

3. He demonstrates an apparent cost bias, particularly for the very highest ratings, where the average single malt sells for an estimated $1400.

4. Last - and I speak as a copywriter - his "reviews" tend to be of a softer variety.

Now in truth, and if I had the time I could probably find a way to adjust his ratings (I did for the other reviewer I mentioned), to find the real cut points between bad - average - good - great, but that of course is hardly likely. Why?

He's committed to his path, and to his self-image as tough and fair. He's likely making a decent living, and carries the approval of most of the distillers with whom he necessarily maintains a symbiotic relationship. His formula works for him, but with all due respect, doesn't for me.

In closing does that mean we should ignore his reviews? Not at all, his tastes and your own may be simpatico - just ignore the scores. Personally I'd deduct at least 5 for scores over 95, and 10 points for those in the 80's and lower 90's.

Again, thanks for your views.


*******
Late Addition:! Hansell makes a very big deal out of his panel of internationally known reviewers, including Dave Broom, Gavin Smith, Dominic Roskrow and Lew Bryson. Hell, it impressed me! I just now revisted the Advocate to find out how many reviews these illustrious reviewers had provided? Answer...

Exactly one! By Dominic Roskrow for Compass Box Great King Street. And this out of over 1200 reviews. Does Hansell talk the talk? You bet. But...

Correction!
As I do, I revisted the Advocate to read a few reviews, when I discovered few reviews by Broom and Smith, so I rechecked their handy search function. My original search was dead wrong - I'd guess I accidentally narrowed the search. The actual numbers: Broom, Roskrow and Smith published about 200 reviews (out of 1258, about 17%). But by far the lion's share of reviewing is done by Hansell, about 1058 reviews.

My apologies. But I'm still mostly (80.3%) right, lol...
Last edited by Capn Jimbo on Sun Dec 11, 2011 1:13 pm, edited 7 times in total.
User avatar
Uisge
Cap'n
Posts: 178
Joined: Sat Nov 05, 2011 4:32 am
Location: Marvelous Madera Ranchos, CA

Post by Uisge »

Good points you put forth, Cap'n.

Thanks for your reply. Now I know that you aren't the kind of captain who sez "The beatings will continue until morale improves!" :wink: :lol:
User avatar
Capn Jimbo
Rum Evangelisti and Compleat Idiot
Posts: 3550
Joined: Mon Dec 11, 2006 3:53 pm
Location: Paradise: Fort Lauderdale of course...
Contact:

Re: Late breaking news!

Post by Capn Jimbo »

Late Breaking News! The Advocates' Reply!


And news it is. Seems as though one our valued members claims to have contacted the Advocate. This member further advised that the Advocate's defense amounted to this:
...liquors which rate in the 60 and 70 in general do not rate being published, and even then he doesn't sample ALL whiskies available
This is a variation on the common copout which is "...but I only pick the best spirits, so naturally my scores are higher...". That has been well addressed in in the Scuttlebutt section, "Normality and Idiocy: Bell Curves 'n Rum Reviewers!", subject bell curves and normal distributions, discussed (here).

This variation is "...oh, I actually did sample and rate a bunch of 60's and 70's, but they weren't worth publishing". OK fine, let's take him at his word, and examine this.

The basic question is: just how many spirits would he have had to sample and score, and then reject, to justify the high scores he did report?


The Answer

Sticking to what we know about:

1. He did sample, taste and score about 400 exceptional (90-100) whiskies and 800 great (80-89) whiskies.

2. Based on the his 400 reviews in the 90's, a completely normal distribution would require:

90-100: 400 spirits (actual)
80-89: 2,800 (expected, 800 actual)
70-79: 13,600 (expected, 97 actual)
60-69: 2,800 (expected, 2 actual)
<60: 400 (ibid)

Total: a minimum of about 20,000 whiskies must have been tasted and scored to insure a normal and credible distribution. Mind you, no sampling results in a perfect distribution, but to be considered credible, his numbers must approximate these.

Otherwise there is bias in either the protocol or in the reviews themselves, causing a tilt to higher scores. The fact is this...

Since he published only 1258 reviews, he'd have had to sample, taste and score about 18,700 whiskies and having done the work, then to exclude all of them from his reviews! This would also include excluding about 2000 "great" whiskies scoring 80-89. Do you believe he sampled and scored all these, yet excluded them (particularly the "great" ones in the 80's)?

That's exactly what is being suggested. If he did not sample and score that many whiskies, the distribution skews high.


Another view

Let's compare just the top two published categories alone - 400 in the 90's and the 800 in the 80's. Again based on a normal distribution, 400 "90's" would require 2,800 in the "80's", or conversely, 800 "80's" would justify only about 100 scores in the "90's".

If the former is true, he'd have to have reviewed and scored 2,800 spirits scoring in the 80's - but - he only reported 800 of them? This defeats his claim that he publishes the better spirits. I can't explain why he'd report only 800 but exclude about 2,000 with the same scores. Can you?

Of course not.

Now if the latter is true (that he tasted only 800 spirits in the 80's), then his 400, 90-100 scoring spirits is way, way overrepresented. Simply and again to maintain a credible distribution - 800 scores in the 80's would justify only about 100 scoring in the 90's. Yet he reported 400 - nearly four times the number expected for an unbiased result.

Here, there's only two possibilities:

1. Either he failed to report about 2,000 reviews in the 80's, or
2. He published four times the number of 90's expected.

Personally, I'm not buying the former, and believe the latter.


Bottom line

The Advocate's reported counterclaim is really valid and his scoring unbiased if - and only if...

1. He sampled and scored 20,000 spirits, but reported only 1258 of them.

2. He had even more spirits that he didn't sample. Wow!

3. He excluded about 18,700 spirits, including about 2,000 scoring in the respected 80's.

4. An examination of just the 80's and 90's he reported - alone - shows a huge imbalance in favor of the 90's.

Honestly now, open your eyes and minds and be really, really honest. Do you really, honestly, rationally believe that anywhere near 20,000 spirits were tasted and scored, but mostly excluded? Do you believe he even has that many spirits (and more that he hasn't sampled)?

If you do, then his reported scores are perfectly fine, and are really quite normal and we can stop right here. But then consider just a few more things...

1. The site claims "Due to the increasing amount of new whisky releases worldwide, Whisky Advocate has expanded its number of Buying Guide reviewers to five people: Dave Broom, Dominic Roskrow, Lew Bryson, Gavin Smith, and John Hansell."

Yet of the 1258 published reviews, only about 200 were reported as by reviewers other than Hansell himself. Oops.

2. Speaking of Dave Broom, we analysed all his many rum reviews from his book "Rum". Result: a perfectly normal distribution. Broom reports em all - the good, the bad and the ugly.

3. F. Paul Pacult is considered perhaps the most prolific publisher of spirits reviews anywhere. His recent encyclopedia of reviews, of all spirits ("Kindred Spirits 2") is massive, but contains a little over 2400 reviews. He also reports all scores.

4. Jim Murray in his massive "Whiskey Bible, 2012", reports only 1000 whiskies, and again, reports all scores. Mind you this book has been printed and expanded for years.

5. Everyone knows the Malt Maniacs. In ten long years Johannes has managed to review but 2000 whiskies. And his famous cohort, Serge (of Whiskyfun.com), despite being the maddest of all, has topped out at a current 7,534 reviews. And both of them too publish every single review, every single score.

6. Now consider a much more common beverage, wine. The king of wine is the well known Robert Parker. One of the biggest books you can find on the shelves is Parker's huge "Parker's Wine Buyer's Guide, 7th Edition". This humongous book is 1,536 pages, simply massive! Still, he reports only 8,000 wines, and he too reports all scores.

Remember, there are far more wines than spirits. So I'll ask you again - do you honestly believe the Advocate has tasted and scored about 20,000 spirits (not wines and beers) - and - withheld 18,700 of them, including 2,000 highly rated spirits?

Do you? Do you really? You decide... and then tell me.
User avatar
The Black Tot
Admiral
Posts: 282
Joined: Thu Aug 21, 2014 6:45 pm
Location: Houston TX and Caterham, UK

Post by The Black Tot »

What is a "shelf talker?"
User avatar
Capn Jimbo
Rum Evangelisti and Compleat Idiot
Posts: 3550
Joined: Mon Dec 11, 2006 3:53 pm
Location: Paradise: Fort Lauderdale of course...
Contact:

Post by Capn Jimbo »

A shelf talker...


Is like a bottle hanger, a small promotional card that can be affixed to a shelf to draw attention to that particular bottle. These are often provided by the distributors, or distillers, but in the case of the Advocate is a printable file that features their review and high score and can be printed out.
User avatar
The Black Tot
Admiral
Posts: 282
Joined: Thu Aug 21, 2014 6:45 pm
Location: Houston TX and Caterham, UK

Post by The Black Tot »

Aha. For the ambitious liquor store owner who wants help from Whisky Asskissate (too rough?) to push the product by sticking it on the shelf.

I see.
Post Reply