Some years ago it became apparent that we all were at the mercy of any rummie come lately who had enough computer skills to toss up a website and hold forth as a "reviewer". Some even had the moose balls to name their own "methods" - before they'd had any experience, and worse yet, seemingly possessed of a remarkable, hypertaster's bitter palate.
The Artic Wolf was first reviewed in January, 2011, and summarized by this distribution of scores...
As should be obvious, there was a considerable bias toward the higher scores, with only three rums scoring below "average", but with nearly half his reviews receiving very high scores. This was hardly realistic, but to be fair, only 55 rums were scored. Although this is still a reasonable sample, it is barely so. What to do?
Wait, and wait. Which we did, and now this amazing, sefl-proclaimed hobbyist (who once denied he had any commercial intentions), has clearly gone to the dark side and has now reviewed an amazing 371 spirits, not to mention an estimated 500 or 600 drinks recipes in little more than a year. No wonder he has a good layer of winter fat.
Let's see what happened. Did the Wolf become more or less biased?
Survey sez: More biased. Seriiously check the two charts - his shift to the right - to the very highest scores - should be obvious. This time I superimposed what a normal distribution would look like. Here's the actual numbers:
50: 0 spirits
55: 0
60: 0
65: 3
70: 11
75: 38
80: 85
85: 119
90: 93
95: 22
_______
Total spirits: 371
Remember, a score of "75" should be both the median and average score in a normal distribution. In this case only 14 rums score below average, while about 280 score above! The Wolf's median is in the upper 80's and over "85" - these in an range that most top reviewers would give 4 stars (out of 5) for "great" spirits. Mr. Wolf places 204 of his reviews in this 4 Star category. Worse yet, in the 90 to 100 category, what top reviewers would call 5 Star spirits, Mr. Wolf has awarded this top honor to an unbelievable 115 spirits. Holy Mooseshit, Batman! That's a total of 319 rums in the "great" or "superlative" classes.
As for "Average: spirits, there are only, gulp, 49 spirits, and for "below Average"? Only 3.
What does this mean?
If a gatherer of data, in this case a reviewer is unbiased, the "Average" or 3 Star category should be the largest. The number of spirits below average should match the number above average. In Mr. Wolf's case, the number below is 14, while the number above is 319! Talk about imbalance. The same is true for the top and bottom ratings - there should be very few of each (typically about 2.1% of the data). In Mr. Wolf's case the number of bottom rated sprits is zero - none - while at the tippy, tippy top ("95+") he reports a stunning 23.
Let's put this in further perspective by comparing these numbers to a normal bell curve and distribution:
Normal Distribution:
50-59: 8 spirits
60-69: 50
70-79: 253
80-89: 50
90-99: 8
Mr. Wolf's Distribution:
50-59: 0 spirits
60-69: 3
70-79: 49
80-89: 253
90-99: 115
Folks, this can't be blamed on sample size, which is more than adequate. Nor can it this be attributed to the common response that "...I only pick the best spirits". First, Mr. Wolf appears to review anything that is gifted to him for review, and second it doesn't matter anyway. Here's why...
It's like throwing darts. It matters not where you aim at the target - your darts will accumulate in a normal fashion - with a few hitting the aiming point, a few missing wildly, and most clustered in an "average" distance away. This pattern is common and applies to data collected for any endeavor from shoe size, to human height or weight, number of tomatoes on a plant, to spirits reviews. That is simply life. Trust me, the top reviewers (think Dave Broom) show perfectly normal distribution of scores.
So the question is before us - was Mr. Wolf biased, and has it changed for the better or worse? You decide...
