Here's a little story I like to tell about the second ever Roasters Guild Retreat in 2002. During one of the first sessions at the retreat there was a cupping. When the group that was participating was asked who had ever cupped before, more than 80% of the crowd indicated that they had not. Nowadays, just 10 extremely short seeming years later there is cupping from top to bottom across the whole specialty coffee industry, or at least people regularly look at coffees though the lens of a cupping-like activity.
I think that it's great that more and more people can use this tool. I've put a considerable amount of energy into teaching people how to do it, and many of my most memorable coffee experiences have been at the cupping table. But I think that the tool itself is frequently used improperly. Plus I feel that we are perhaps over-reliant on the practice as an evaluation tool. All of this leads to a rather narrow perspective on what the potential for any coffee can be. I think that in order to get any kind of wider perspective on any particular coffee then you have to look at it in more than one way. Not just with different brewing parameters, but also looking at different roast profiles of it.
For the purposes of evaluating a coffee for purchase, or just looking at a coffee that you've already purchased and deciding what you're going to do with it, I strongly feel that you have to look at more than one roast of it. Now, do you need to get crazy and look at every single possibility to get a total potential perspective? No, but in looking at at least 2 different roasts of any particular coffee can give you a much clearer idea of what the coffee can do, and 3 even more so.
I was very fortunate last year to work with Paul Songer of Songer and Associates in developing a class on sensory test design for the Roasters Guild Certification Program. If you don't know who Paul Songer is, he started in coffee 25 years ago at Allegro Coffee Company and is the Lead Quality Control and Technical Advisor at CoE, Alliance for Coffee Excellence, Inc., as well as having worked on a number of other projects in coffee producing areas helping to build specialty coffee production there, including extensive work in Rwanda. Songer has thoroughly studied and practiced sensory evaluation.
One thing that stood out to me while working on this class was the complexities involved in analyzing the data collected through cupping, especially when looking at a panel. A lot of that has to do with putting a panel together; what their experience levels are, how calibrated they are as a group, adjusting for high and low scores, etc. etc. The next layer of difficulty is in the design of the test itself; what are you looking for? Is this simply a preferential test, or is it to determine a noticeable difference, what kind of controls have you set in place so that you can be sure that you're looking at/for what you intend? Listening to Paul talk about how they look at the results from Cup of Excellence judging panels and looking at some of the charts that broke the data down was dizzying.
All of this can be quite the rabbit hole to jump into. I do think that you can casually look at some coffees without setting up a totally sterile environment or without being a statistics wizard, but I also think that; 1. you need to be honest with yourself about your abilities and those of your panel on any given day, 2. you need to have a clear idea of what you're looking for and at on the table, and 3. it would be considerably helpful for you to look at the coffees a couple different times and in different iterations before making a final judgment. Something that struck me when we actually gave the class at the Roasters Guild Retreat in Stonewall last August was what one of the participants said during one of the group sessions in the class talking about how they set up their regular cuppings to set production roast levels. What he said was that they used to cup all of their coffees at different roast levels but that the lighter roasts always won, so they stopped cupping them and just started roasting everything lighter.
Well, of course the lighter roasts always won. They were only looking at the coffees in a way where the positive attributes of the lighter roasts are better highlighted and the negative attributes of the darker roasts are amplified. A big problem here is that for the most part their customers aren't cupping the coffee. They're basing the entirety of their evaluation on an experience that few others outside of themselves will have with the coffee. But is there also something here about what we're looking for in a cupping in the first place? What have we decided is a positive attribute and a negative one? If there isn't any "fruitiness" present do we actually reject it?
Cupping is mostly designed to look at whether a coffee has any defects, is consistent from cup to cup, and that there is sweetness present. Listing 10 different fruit flavors is not the main focus of a cupping. I also think that there are some problems with taking a qualitative measurement of both acidity and body, two things that can be so greatly influenced by roast development, unless you are looking at a couple different roasts of that coffee. I don't have any problems with forms. Forms are great for capturing data, it's cuppers themselves and their test designs which cause the biggest problems in my book.
So what's the big problem? To paraphrase what I said in the close to part 1 of this rant, the problem is that great coffee can be great for so many reasons. When we only look at coffees in one way, we greatly reduce what we can do with the coffees as well as what experiences we get to share. I can see why that's attractive to some people. There are businesses that want you to be able to go into a store in Cincinnati and have an identical experience to the one that you would get in Anaheim or Singapore or Oslo. It just isn't that special.