Numbers don’t mean anything if they aren’t defined and put in perspective. A comment raised in my recent post here shows that I need to post my rating scheme, which you will find in Malt Advocate magazine. (The reviews here on WDJK essentially are advanced reviews of what will be published in Malt Advocate’s Buyer’s Guide.)
RATING SCHEME
95-100
A classic! All components are balanced appropriately, with the complexity and character expected in a classic.
90-94
Outstanding! One of the best for style. Distinctive.
80-89
Good to very good. Plenty of character and no identifiable flaws. Worth seeking out.
70-79
Average. No unique qualities. Flaws possible.
60-69
Below average. Major flaws. Avoid.
So, when I review a whisky, keep this rating scheme in mind. For example, when I give a whisky a rating in the low 80s, this doesn’t mean I don’t like it or think something is wrong with it.
I wonder why all Whisky reviewers use such a narrow, and high range, to note. IMHO, when issuing a note out of 100, “average” should be 50. There should be very few in the 80′s and almost none in the 90′s.
This is a pretty standard rating scheme, and the range (essentially 60-100) is certainly broad enough. Many would argue, Oliver, that it’s too broad.
Even as it is, I have to keep reminding people that a whisky that a whisky rated in the low 80s is still a nice whisky. Many consumers (and especially producers) think it is a kiss of death or something. It isn’t.
Olivier, you have to keep in mind that the 100 point scale has to encompass anything from totally disgusting and nauseating to absolute perfection. A rating of 50 is actually “average” on this absolute scale, in a way that anything below this line can ce called bad. What often causes confusion is that the quality of an average whisky is in fact pretty good, so it is a general agreement that a fair average dram should score about 75. Imagine overall whisky quality going up even more. Then an “average” whisky might score 80 or 85 on an absolute scale. You always need a calibration point to know what “average” exactly means.
It’s nice to see that John’s scale is basically congruent to my own. Here are my personal definitions:
0 to 5: Don’t swallow this stuff. Spit it out!
6 to 15: Be courageous, finish it!
16 to 25: Can there really be someone who actually likes this?
26 to 35: Slowly approaching whisky territory
36 to 50: Some might even like it, but there are too many flaws
51 to 65: Drinkable but soon forgotten
66 to 75: Has its good points but also weaknesses
76 to 85: A good dram but there is still room for improvement
86 to 95: Something not to just drink but to celebrate
96 to 99: Approaching perfection
100: Do we really want something that cannot be improved?
And speaking of overly high ratings, you can read my take on Jim Murray’s “Whisky Bible” on the first page of my blog (just klick on my name)
I like this system. The three lowest scores are pretty funny.
I’ve never understood the need for a 0-100 scale if nothing ever rates below 60.
If there is a 100, there has to be a 0 as well. Although the nether regions of the scale are rarely ever used, they have to be defined in order to get the calibration right. Almost all whiskies are at least somewhat decent, so ratings will usually be above 50. But there really are whiskies that score very badly. Loch Dhu springs into mind, and I recently tried the German “Black Forest” single malt that I gave a score of 20. I might have been a bit over-critical, but it really tasted awful.
I like your scale because it’s similar to the one I use. As you say most whiskies are decent so therefore will score at least a 50. “Drinkable but soon forgotten” or as I put it “Could take it or leave it.”
I’ve not had the (dis)-pleasure of trying Loch-Dhu or the Black Forest but I’ve tasted some truly horrible (to me anyway) stuff and have written in my notes, “Why would anyone voluntarily drink this?” on more than one occasion.
So far I haven’t tried the Loch Dhu myself yet, but I most probably will get a sample very soon. I’d never thought I would ever get a chance to try it. I am actually looking forward to trying it, well knowing what to expect
From what I recall, it tasted as much like rum or bourbon as it did scotch. I think it was designed to compete as a mixer.
The 0-100 is probably common due to the fact that so many people are familiar with it – spending years upon years being judged via this system (school). An F is an F, whether it is 55 or 5. In a “curved” environment, C is the average, which is 70 or 75, not 50. It’s not a true 0-100 point scale, IMO.
Maybe it is flawed, but its been around and is easily comparable.
John, I know you have only the best intentions, but a rating in the low 80′s can definetly be a kiss of death– Especially when the review describes a counterfactual version of the whisky, which would have gotten a higher score (if it had been non chill filtered… if it had no coloring…. if it had been so many years older… etc…).
Red, when I review a whisky, I try to think of what could be done to make the whisky better. I believe that’s called constructive criticism (not a kiss of death). Given that you are probably the most critical commenter here on WDJK, if anyone should understand my nature of evaluating, you should. (And I know that you also only have the best intentions.
)
In theory, unless I rate a whisky a classic (i.e., 95 or higher), there’s something I can say about the whisky that I would like to see improved.
And you are only mentioning the critical components of my reviews, not my positive comments. Indeed, I always try to be fair and show both sides. The less I like a whisky, the more negative comments there will be; the more I like a whisky, the more positive comments there will be. Even with the whisky you use as an example, there are many positive comments in my review.
Sadly, I suppose that in some ways, you are right. Companies usually don’t like a rating in the low 80s, even though to me this means it’s still a pleasant whisky. I don’t think I have anywhere near the power influence to create a “kiss of death” though.
Of course you could just report on what the whisky’s like to you– no number, no constructive criticisms. But that would peel away many important layers of what makes you relevant and interesting to the whisky community– Plus there are people all over the net who do nothing more than post tasting notes… So I think the best thing would be to present your reviews in a public forum and be willing and ready to further explain them
I have to agree with Red on that one. It’s the same phenomenon with wine right now. People look for 90 points or higher. Anything else just gets passed on. That’s not the fault of the critics necessarily. But I wonder how many people actually read the reviews and get past the number.
I’ve mentioned this with wine before but those that avidly drink wine in my circle hardly ever look at the rating when buying, especially from some critics. I realize a lot of novices do refer to a rating system which is a harsh reality for those selling wine. In fact, I can think of one critic that I steer clear of entirely when he gives a high rating. It’s not that this critic doesn’t know what he’s talking about but rather our palates do not mesh. In the end you simply need to find the critic that suits your palate and run with them. For wine I seem to have good luck with Wine Enthusiast.
My personal experience with John is that our palates are very similar which is why I follow and respect his opinion(s). Granted, I probably wouldn’t consider anything below 80pts. Anything above 80pts will depend entirely on price and availability. That’s when personal evaluation and decision making in purchasing needs to take over.
I think there are certainly unintended consequences of giving anything a score under 85, but perhaps there’s no better way to judge something based on taste which is subjective. I wouldn’t respect the high scores if there wasn’t a base of 80s sitting there for comparison. I may be willing to shell out $50 for an 85, but I would run to the store to buy a 95 for $30 (EW Single Barrel). Spirits biz folks are obviously aware of this fact. There will always be a tension between companies trying to produce products they are proud of, for a group of drinkers who take pride in authenticity, and corporations who would send John a thousand samples before he said he liked it in order to meet a certain profile. I don’t see this going away anytime soon. Let’s hope a balance can ultimately be struck between great products made with purpose and integrity, and businesses which must make money to keep producing these products. In the end, some amazing whiskey will find its way to the top. Tuthilltown makes decent whiskey which Jim Murray won’t cut any slack — does this mean they should dump their barrels and give up? Hell no.
I have read, and agree with the comment, that the reason so many whiskies get such “high” ratings is because the distillers do their best to put out what they believe to be a quality product at a given price point. Subjectively, we all know that some do better than others.
As John has recommended on an earlier post, I read several reviews, not just his. They are not always in agreement because of the subjectivity. But when they are all in agreement on the quality of a whisk(e)y, that is definately one to try.
Ever since I discovered whisky ratings, i sort of considered them to similar to grades in school. So in that respect, John’s ranges are spot on. And once something has a low enough grade not to bother with, the only thing that really matters is whether it is just not worth drinking, or truly vile and avoid under any circumstances. The only change I might suggest is to break up the 80′s into two ranges, just like the 90′s. There is after all, a big gap between an 81 and 89 rated dram.
Having said that, there still needs to be additional clarification beyond a single number. Something that gets an 86 for example, could be an older whixky that has faded, or a decent younger dram. So the additional notes that the Malt Advocateprovides are perhaps more useful to me, than the numerical rating.
Slainte.
Louis
John,
I’ve spent more time learning your “system” than any other one online or in print, although there is great fun in comparing your ratings to others’. I try to compare and contrast your new ratings with your past, seasoned ratings for whiskies I know. I appreciate your time and dedication to your system, and I don’t waste my time refuting it. As no two palates are identical, we’re not in lock-step in our assessments, but I’ll be danged if you don’t hit about 95% for my taste.
Some of your low-80s ratings have turned out to be some stand-out whiskies in my book.
My only problem is with your “Whisky Reviews” page. I try to research past reviews often, but all too frequently I get a notice that says: “Server Error in ‘/’ Application.” I don’t know if it’s my ISP or your site’s problem. Anyone else see this?
Honestly, I feel the same as you two bit. Almost all of the time John’s ratings are spot on, with the notable exception of a few low 80′s ratings (Bruichladdich 20 3rd edition for example). In fact, the Corry was the first time I have ever seriously differed with John about anything besides that.
I can tell you first hand that ratings sell. It’s the nature of the beast. Reviews in the 80′s may not be the kiss of death, but no distillery wants them. Now imagine getting 82pts AND having a retail price of $100. It doesn’t paint the best picture and sales could suffer.
John, please give the “kiss of death” to more perfectly good whiskies. Maybe the prices will come down.
Actually, you will be seeing a lot of whiskies dropping their price in the next year.
Seth and Cowboy, it certainly is important to take into account what expectations are raised with a whisky. 82 points for a 30yo that costs $300 is certainly a deceiving score, but for a $30 supermarket bottle it’s not a bad mark at all.
Except for the fact that to 99% of people in the world, 82 pts for a $30 whiskey means you get what you pay for, and 82 pts for a $300 bottle means you don’t.
This is all most interesting, but is it esoteric?
Do we really believe that the “nose” at a particular distillery doesn’t know that his or her $300 whisky will only get an 82 rating by the critics? I can’t believe that the Master Distillers or Malt Masters or any of a myriad other titles aren’t able to judge their own whisky by the same standard as John or Serge or ….
I’m sure that the “nose” at most if not all distilleries can judge their own whiskies. However that’s not what they’re in business to do. They’re in the business to sell whisky. It’s why I usually don’t like it when distillers include their own tasting notes with their whiskies. You could make an analogy with Hollywood. They’re not going to tell you a movie is bad, they want you to pay to go see it. A distiller isn’t going to point out flaws in their whisky, they want you to buy it. It’s why websites, books, and blogs like this exist.
I second that emotion!
John, point taken with regard to that recent review in particular, and more generally as well. As a person who spent years grading the work of college students, I agree that the 60-100 scale has lots of room to move and that 80-85 is typically the B- to B range, which (unless you’re from, say, Yale) means “good.” [As an aside, one of my favorite statements from a Yale student to whose paper I'd given a 'C': "But Mark, don't you understand, we get B's here for turning it IN."] Back to the scale, and we can realize that low 60′s are low D’s. Your scale makes excellent sense from that perspective.
For me, with that particular review, it’s a matter of (a) already doubting the producer and (b) liking the young Islays I’ve had enough to stick with those I trust. If the rating had been higher, then I would have had reason to re-evaluate. I’ll try it at the Fest. Open-mindedness is a virtue, right?
One more thing, John: Have you, would you, could you review Finlaggan?
I’ve only just started reading your blog, so thank you for explaining your rating categories.
I appreciate the kind words, everyone, about trusting my reviews. I am human and makes mistakes, but I have been reviewing whiskies formally now for almost 20 years and really try to do my best with each review. One of the things I think that helps is that I always try to taste a whisky twice before posting a review. And on one of those occassions, I swallow.
If 98%+ of whiskies score 60 – 100, why not go with letter grading? That is a scheme familiar to most everyone and is a bit tighter. Regardless of the system used, the best proof is in the sampling.
John, I would appreciate it if you could clarify this. I have always taken your marks to indicate what you would rate a whisk(e)y amongst it’s own kind..kind of like Ralfy does with his Malt Marks, Blend Marks..etc.
So when you give a bourbon a 96 and you give a single malt a 86 you aren’t saying necessarily (although it could be the case) that the bourbon is a lot better than the single malt..you are just saying that as bourbon goes, this one is a 96….right??
The old apples-and-oranges thing…Murray says his scores are absolute, that a 90 blend is better than an 89 bourbon which is better than an 88 malt. You’d have to say that, if you’re claiming your scoring is in any way objective. I don’t think there’s any such thing as objective scoring, and I wouldn’t really be interested in it if there was–the enjoyment of whisky (and food and art and literature and film) is an inherently subjective thing–we each have our own tastes and criteria for excellence. Very few people will claim to be able to put malts, blends, Irish, bourbon, etc all in the same basket–most of us have a taste for one or another over the others. Actually, I don’t pay much attention to scoring, anyway, except in a very broad “I like this, I don’t like that” sense. I’ve generally found that most of the whiskies I enjoy most score somewhere in the 80′s; those scored in the 90′s may or may not float my boat, and I inevitably find myself thinking, “What’s the big deal?”
I see absolutely no point in having different scales for different drinks. And I am pretty sure that John does not rate bourbons differently from scotches.
And as much as I love Ralfy’s reviews and acknowledge his knowledge about whisky (and it’s certainly better than mine), I can’t get around that “malt mark” and “blend mark” thing. How does an 80 point malt compare to an 80 point blend? Chances are that the blend would rate significantly lower, if the same scale would be used for both. But there are some old top blends that taste better than 95 pt malts. What would be their “blend mark”? 110? Where do we draw the line? A different scale just because there is grain involved? How to rate a single grain or a vatted grain? Do we need separate scales for ryes or wheated bourbons?
There are so many different kinds of whisky that it makes no sense to introduce separate rating scales, in my opinion. There would always be the need to explain just how they differ.
Texas, my ratings are absolute. Something rated in the 80s is “Good to very good. Plenty of character and no identifiable flaws. Worth seeking out.” It doesn’t matter whether it’s a bourbon, blended scotch, Japanese single malt, etc.
I enjoy, and have respect for, all whisky categories. I think that any whisky writer who reviews and rates all whiskies must be this way. There can be no personal biases.
The issue here is, most whisky consumers usually DON’T like all categories equally. So it’s easy for someone to say something like: “How could he give that XXX single malt scotch and XXX bourbon both a 93. Clearly the single malt is better.” But the main reason why the person feels that way is because they have a bias towards single malt scotch.
I hope this helps to clarify my logic.
Thanks, makes sense.
And obviously the reader/consumer will filter it all through his own biases. If you simply don’t like bourbon, for example, it won’t matter to you if one rates a 95 or a 65.
I agree absolutely with your appraisal, John. A respected reviewer MUST be objective despite the category being reviewed, yet I have no choice but to be biased BECAUSE OF my personal preferences for a given category of spirit. Very well put!
I have mentioned before that my initial attraction to Malt Advocate was the respect you afforded every individual style of whiskey. Having progressed to a critical appreciation of whiskey from the beer world, I had been put off by the predisposition of beer reviewers to hold in disdain regional beers that did not meet their preconceived parameters for craft status, despite being perfectly good products in their own right.
I find this to be one of the main differences between the work of respected whiskey reviewers and that of well-regarded beer pundits to this day, and it makes all the difference in the world to me.
Your rating system hasn’t failed me yet and I do look at it like a school grading system anyway. Not hard to convert an 80% to a B- grade and so forth. Theoretically speaking, the need to have anything below 50pts needs to be an option assuming John ever gets a sample of whiskey distilled from feed corn and aged in an oak horse trough for two years… now that’s the review I would love to see!
Heck, I might even go out of my way to purchase a cheap 40pt whiskey just to see what it was like!
I assume you’re not likely to get something this terrible considering the modern age of distilling and (hopefully) quality control. But hey, you never know when that new kid on the block pops his ugly head out with a terrible dram. That’s when the “kiss of death” would be totally justified.
I think that Chef! has a couple of great points about the wine world and ratings. Avid wine drinkers and scotch consumers may find that one reviewer seems to mirror their palate and mostly stick with buying according to their reviews. But just the fact that John scores something in the 60′s makes me curious enough to get ahold of a try it, it’s more of a sliding scale as I learn about my own palate.
Those just getting in to whisky or wine will more often stick by a reviewer and only purchase what is affordable and rated in the 90′s, always seems like a good value. If I give a Higland Park 18 to someone who’s never tried single malt before, and let them know how it scores, Piccault’s comments, etc. They will inevitably “like” it because they feel like they are supposed to. Also, it’s just a really great single malt!
I third Ralph and Seth’s comments about distillers and their own tasting notes. I think that they are accurate the majority of the time, but are ultimately written to help sell the contents of the bottle. Also, they will sway what some are actually tasting, and limit what others taste. Newer consumers have to trust the experts…
In my experience it’s the newbies that are more objective. I’ve worked with some true wine snobs in my industry and I love to prank them around Christmas time by refilling high priced bottles that I’ve stashed away with a decent bargain bottles of red or white table wine. Gets’em every time! I think the branding and marketing has a lot to do with that and I think we’re all victim to this at some level as we become more exposed.
Try doing a whisky tasting with someone that’s never been exposed to good whisky and you’ll be surprised. I did a tasting for my Mom and cousin recently with twenty different types ranging from all over the world and with huge price differences. Trick is not disclose prices, scores or favorites until the end. It was interesting seeing what they leaned toward. My mom had me buy a bottle of Yamazaki 18 on my next trip to the store and loved all the peated versions I had open. I see where I got my palate now
I would give this thread an 88 – Good but not great- Some complexity and not too woody – Monique provided a strong finish
And would you say that your comment and this reply are adding length, but no real nuances, to that finish?
Red_A:
And this one, too. I detect a hint of smoke. Rather like the steam climbing from the cow pattie the sun just hit on this -13 degree morning.
A practical suggestion which I see no one has made yet so let me: could you put a link to your rating system “decoder” someplace prominent on the site (in the top nav bar for example) and maybe even starting linking your ratings to it? Debate on we may, but being able to know the intent and meaning behind your numbers seems like a generally good idea.
Michael
Michael, if you click on the “Buyer’s Guide” in the top nav bar, you will go to the reviews landing page which outlines my rating scheme.
John, the comments on these pages are timely. We’re a small family Australian retailer, and as such, we’ve had to put some thought into the issues surrounding the numerical rating of wines and spirits. Put simply, our customers have demanded ratings for the products we stock and recommend, but they also want a degree of transparency and accountability. They want to know exactly wh one wine scores 89 points and another 90? Was it the palate or aroma or some other aspect that lost a point? Recently we’ve developed an approach to the rating of spirits, based on a successful parent system for wine. We’ve been following your web site for some time and as someone who’s evidently had great experience in the world of whisky, we would certainly appreciate any feedback or criticisms on this project. There’s a short paper on the subject here:
http://www.nicks.com.au/Index.aspx?link_id=76.1676