12 Jul Why It’s Foolish to Rank Breweries
by Mark E. Lasbury for Indiana On Tap
Humans are basically simple animals – our world is so impossibly complex that we look for ways to categorize it, This makes it simpler to understand – but a lot can be lost during the process of categorization. Take Elysia chlorotica for example – it looks like an unremarkable sea slug, until you learn that it can perform photosynthesis and only eats once in its lifetime. So, is it a plant or animal? It’s definitely an animal, but by reducing it to just this you lose a lot of what makes it special.
If one had to rank the list of ways people like to categorize their world and the things in it, creating rankings would be right at the top. Top tens, best ofs, etc. – there are all sorts of ways we try to rank things. I even saw a ranking of fifteen articles that rank things on Google the other day. A “favorites” list admits that it’s subjective, but rankings and “best of” lists suggest that they are somehow objective – they usually aren’t.
Some things are easier to rank than others; the more objective the criteria, the better we can agree on the ranking. Why do you think sports fans memorize all those statistics? It’s so they can justify their rankings to others. But, can you rank home run hitters based solely on the number of dingers they jack out? What about the quality of pitching, the size of their home ballpark, the live ball vs. dead ball eras? Objective isn’t always so objective.
The Best of 2018 was a widely distributed list, but it was really just the editor’s opinions. Do you think he had all the beers made in 2018? image credit: Craft Beer & Brewing
It’s no surprise that the ranking phenomenon has found craft beer as well. We rank the best breweries in the country, the best beers of each style, the top sellers, the best can art….. it goes on and on. Looking for a ranking is a daily occurrence on social media; someone is getting ready to visit a certain place, what are the best breweries there? For better or worse, people often ask Walter and I what are the best breweries in Indiana or in a particular part of the state. I’m here to tell you that it isn’t possible to rank them without a very detailed set of criteria specific to that particular seeker of information.
I understand why people ask, I really do. No one has unlimited time or money, so they can’t try everything out. They seek input to help them simplify their search for good beer. It ain’t a cheap hobby so if one can avoid wasted money on a less than stellar experience, then do it. But can people trust the information they get? There is little in life that’s less objective than whether a craft brewery is good or not.
In their most loathsome form, rankings are nothing more than click bait. It’s in the nature of some groups to solicit their audience for rankings or throw out a top ten list just to get clicks and responses. Besides looking to simplify the world, the other thing that humans really like is expressing an opinion and letting you know why yours is wrong. The age of social media lends itself to both – expressing and arguing. Groups that ask people to rank breweries or publish a “best of” list in a magazine or online don’t really care what the results are – they are looking for feedback and hits to improve their numbers, it’s a simple way to manipulate you into interacting with them, which makes them look better with advertisers or people looking to rank those entities.
Could you perhaps be more vague? At least the destination was named. image credit: Walter
Let’s take a typical example: Joe and Jan are traveling to Fort Wayne for three days and ask Facebook friends and craft beer groups which are the spots they shouldn’t miss. Normally, that’s all the information they give – now how do you put together a list of places they should go, by what you like or what they like? You don’t know what they like, so you fall back on your own opinions. If Jan and Joe asking are just looking for names they might have not found, and they will look them investigate on their own, that’s fine. But I hope they don’t expect to get a list of the breweries they will like the most.
A second example: A magazine publishes the list of the top ten new breweries in Indianapolis. What does new mean? What is their listed criteria for ranking? Who did the ranking? And most important – does it reflect what you think is important? If you can’t vet the first three items, then the last one is a big fat zero.
And yet people rank breweries and expect you to buy in (or not, mostly likely they just want you to respond) and people ask you to rank breweries. When people ask Walter and I, they usually end up wishing they hadn’t because we ask a long series of questions to help determine what THEY think makes a good brewery, and then we answer with some suggestions based on their input. It takes a while. We ask about many criteria, and any one of these could be used to rank a brewery high or low based on what people hold as important, and how they come to their decisions. Let’s take a look at some criteria people could and do use and show why almost all of them don’t help much in rankings:
Beer. OK, so what kind of beer? Do you think you are capable of rating a brewery’s beer based on objective criteria like technical flaws or style guides – few people are. Do you rate styles you like high and styles you don’t like low?
Are you a BJCP judge? If not, say what you like or don’t, not what’s good or bad. image credit: All About Beer
Beyond this, do you rank a brewery based on how many styles they make or how often they have something new on the list? Are you a person who likes classic styles or the hottest new thing? All these issues will affect a ranking, and all are basically subjective except for comparing to style guides and technical flaws, and almost everyone fall short in being able to assess those. Yes, you can come up with a ranking of breweries, and that list is perfect……for people who think exactly like you do.
Food. Most breweries now have at least some food. And they probably should – people who have something to eat are more likely to stick around for another beer. But do you rank breweries based on whether they have food, who much you like the food, what kind of food, how much the food and/or beer costs ease of getting food, number of choices, vegan or vegetarian options?
Add food to beer, and now you have two subjective issues that you have to parse. How much weight do you give one over the other? Some people don’t care at all about food at a brewery, while other people won’t consider visiting a brewery without food. Is your ranking getting harder to interpret or create now? We’re just getting started.
Medals/Awards. There’s absolutely nothing wrong with raising a brewery in your estimation based on awards they win. It’s probably the closest thing to an objective measure that breweries can get. BUT REMEMBER, a medal is only about the beer; it doesn’t take the other criteria here into account. In addition, you have to take care not to penalize breweries for not winning medals.
Medals can mean alot, as long as you don’t have them mean everything. image credit: GABF
Only a small percentage of breweries submit beers for competitions, and even fewer do it on a regular basis. Therefore, many breweries with no medals have great beers. Are you or your website or magazine going to rank them lower simply because someone else has medals. On the other hand, competitions can really help a brewery make better beer if they use the feedback they get from the judges. And all the entries get feedback, so even though they may not have medals, they could be learning to make great beer.
Atmosphere and amenities. This might not be the most important criteria for ranking breweries, but it probably comes into play more often than not. Some people search out breweries that allow children and/or dogs; others avoid them like the plague. Some breweries have many TVs, while others want people to talk to one another. There’s loud music, no music, old music, new music…. The possibilities are endless and all will be reflected in someone’s ranking of a particular brewery.
Likewise, the experience a regular has is most likely going to be different than what a first time visitor has. The regular knows everyone, has a history with them, is comfortable, is probably a mug club member and gets a better value for their dollar. The atmosphere for one person doesn’t equal the atmosphere for another. Can you rate one brewery high simply because it’s your regular spot?
The people. There are some breweries that get recommended because someone knows the owner, or because it is their own brewery. I know that I have a particular fondness for a brewery or two that has little or nothing to do with the beer, food, or atmosphere – I just like the people who run it and their story.
Good People Brewing in Birmingham is well named – they have good people. image credit: Good People
On the other hand, there is the crowd to consider. Different brewery taprooms have different vibes, and this is often a result of having different crowds. Inn general, people are all the same, except when they congregate around a common location. Then they start to take on a personality of their own. The brewery itself is often a player in this; some places cultivate a more inviting vibe, while others are more regular-driven. Is that going to matter to a website or magazine looking to list the top ten breweries – unlikely. But you may miss something great or fall into a trap because of it.
Ease of access. Are some breweries “better” because they put out more beer and you can find their products everywhere? Or, are some breweries going to be considered chic because you can only get beer at their location or at certain releases? In truth, probably neither matter. But this isn’t a truth kind of deal; as with all these other categories, it’s a subjective, bring your own opinions and baggage kind of deal.
Ratings and Reviews. UnTappd, BeerAdvocate, and that other app from AB InBev hold strong sway when people are looking for places to visit. However, the problems we mentioned in the “Beers” category above apply here. Do you really think you’re getting an objective score on a beer from everyone who has offered their input? Strength in numbers, yes, but have you noticed how many beers hover around the 3.5-4.0 range?
Unless, you find a brewery which scores in the 4.25-4.5 range on most of their beers, you aren’t really learning much. And even at that, how many scores is that anyway? Looking at the brewery score won’t tell you how many people are rating them, or if the brewer’s family is rating every beer a 5. Are you getting objective data?
Not every brewery wants to be national, but does it affect how they are seen and ranked? image credit: Chicagoist
The same caveat goes with looking at Yelp or other reviews. Go ahead and ask breweries how demonstrably inaccurate these are. Of course you won’t know what falsehoods are being put forth unless you know the brewery already. Ratings and reviews are perhaps the only things more subjective than how good the beer is. Consider this – for every category I’ve listed, do you think that the kind of day a person is having ISN’T going to factor into their memory of the experience? Still think that ratings/rankings are worth much?
Conclusion. I hope this has convinced you that when you are asked to say who is best in a certain place or read a top ten list of breweries for a city, you are really getting a bunch of opinions based on the specific things that are almost entirely subjective. If a particular set of ranks doesn’t enumerate the criteria they looked at when ranking and if you don’t agree that their set of criteria are at least partially objective – then what good is the ranking? If you don’t ask a person what they hold dear when they ask you for recommendations, they are you giving them anything close to helpful?
Some people will say that it doesn’t hurt to ask for input from lots of people. The more people that mention a particular brewery, the more likely you are to like them too. Maybe – or is it a self-fulfilling prophecy? More than a few breweries survive on a caché or reputation that may be unearned or past relevancy. Will you like because you think you’re supposed to like it – all of those people loved it, so is there something wrong with you if you don’t?
The best advice I can think of – find a close friend who knows what you like, travels extensively, and drink lots of beer. They’re going to give you a list you can have faith in.
No Comments