The Agony and Ecstasy of Beer Reviews
From the Thoughtful to the Troll, Every Beer Gets an Opinion
When Scott Smith, founder and owner of East End Brewing in Pittsburgh, started selling his beer more than a decade ago, rating websites like BeerAdvocate and RateBeer were valuable resources for his young brewery. In the pre-social-media era, the forums on the sites provided insightful access to consumers, and reviews provided a “first line” for feedback about inconsistencies and quality issues.
As East End has grown and established quality-control procedures in-house, and now that customers can interact directly with the brewery via Twitter and Facebook, he doesn’t check rating websites, which now include Untappd, as frequently for feedback. He also stays away for another reason many brewers can identify with: criticism, gossip, and commentary from the beer Internet’s peanut gallery.
“The beer websites, in particular the forum community on BeerAdvocate, can be pretty toxic, and I use that term with some reluctance, but I find that’s the case,” says Smith. “I see horrible things written about brewers that I know, beers that I know, by random, anonymous people. The Internet is certainly filled with trolls, and it seems like the forums on BeerAdvocate have more than its per capita allotment.”
Scott Smith of East End Brewing. (Photo courtesy East End Brewing)
Indeed, for brewers who have invested significant time, energy and money into making a beer, scrolling through reviews that dismiss it as “meh” and forum threads about overhyped or overrated breweries can be exasperating.
“You sweat and you work and you pour your life into something and then you get like, ‘Eh, I dumped it.’ You get these reviews that have no information; you can’t even really tell if anybody actually tasted it. There’s definitely people posting on there who post a lot more thoughtfully and offer more descriptions, but it’s pretty tough,” says Randy Mosher, author of Tasting Beer and a partner in two Chicago breweries, 5 Rabbit Cerveceria and Forbidden Root. [Mosher is also a columnist for All About Beer Magazine.]
Online beer communities and reviewing sites can also feel like “a detached little bubble compared to the rest of the beer world,” says Mosher, in which imperial IPAs, lambics and barrel-aged stouts reign at the top of the list and members obsess over tracking down rare beers. While he tracks what is being said about his beers online, Mosher says he prefers semi-professional or professional feedback from competitions that are judged blindly or from a more traditional feedback mechanism.
“In a perfect world, we do get feedback from consumers: They buy our beer. They wear our shirts, buy our beer, and we thrive in the marketplace. So that really is the kind of feedback we love to have,” says Mosher.
A Part of the Drinking Experience
Google the name of a beer and review site links will likely show up in the first five search results. The scores are posted on shelf tags at stores like Whole Foods or local beer shops and on beer bar menus. It’s not uncommon to see savvy consumers stand in front of rows of bombers and six-packs, smartphone in hand, checking a beer’s score and scrolling through reviews.
Related: Shelf Talkers Offer More Than Beer Information
Before that beer may even end up on a shelf, the store’s beer buyer might look up the beer online before placing his order with the distributor. And all these numbers are crunched and compiled into lists that are picked up by local and national media, touting the best beer in the world or the best new brewery in the world, drawing beer lovers to seek out beers that everyone’s seemingly talking about.
Consumer beer rating websites have democratized beer reviewing and given license to hundreds of thousands of drinkers to log on, share their opinion and assign a beer a numerical rating. That power has also shifted the relationship between brewery and consumer and changed the discourse of beer culture.
“Beer Drinkers With Opinions”
Todd Alström logged the first online beer review for BrewGuide.com, what would become BeerAdvocate.com, on Aug. 22, 1996. It was for a pint of Berkshire Brewing Co. Steel Rail Extra Pale Ale, served at the Eastside Grill in Northampton, Massachusetts. He rated it 3.85 out of 5 and described it as “a very light ale, with an extremely refreshing amount of carbonation.”
In the 19 years since that first review, 582 other reviewers have logged on to rate Steel Rail Extra Pale Ale, and BeerAdvocate has amassed more than 10 million ratings and reviews. Alström, who was developing video games and tinkering with computers from a young age, and his similarly technically inclined brother Jason Alström originally built the website as a place to post their reviews of beers. They later added forums for users to talk about beer, and in 2000 they rebranded as BeerAdvocate and introduced functionality that allowed the growing user base to review beers by attribute—appearance, aroma, taste and feel—and give the beer a rating.
Todd and Jason Alström, the founders of BeerAdvocate.com. (Photo courtesy BeerAdvocate.com)
Since then, they’ve experimented with attribute weights, flexible rating styles and quick ratings, with single-score “ticking” and no attributes or text. Recently they returned to their roots with attribute-based ratings that “encourage users to think about the beer and share their experience with the rest of the community with a thoughtful review,” says Todd Alström. The rating system in place today is based on the Beer Judge Certification Program (BJCP) score sheet: Appearance is 6 percent, smell is 24 percent, taste is 40 percent, mouthfeel is 10 percent, and overall is 20 percent.
He doesn’t foresee any other major updates to the rating system, but they are highly considering adding some sort of commenting system that would allow breweries or business owners to claim their page and communicate more directly with reviewers. “There are no beer experts, just beer drinkers with opinions’ is an old saying that Jason and I coined and displayed on the site in the early days of introducing user reviews. It was our way of telling our users that their opinion was just as important as the so-called pros, which we still believe to this day,” says Alström via email.
In recent years there have been new entrants to the beer review arena, most notably the mobile app Untappd, which launched in 2010 and counts more than 2.1 millions users. With the software, users have the option to rate a beer from one-half to five stars and can dash off a few notes or thoughts. Check-ins from friends appear in a newsfeed stream, allowing users to see who is drinking what and where and to “toast” it or comment on the check-in. “There has to be some sort of rating system for any type of beer just to know if it’s good or not, just like you have Rotten Tomatoes for movies or IMDB,” says Greg Avola, co-founder of Untappd. “But I firmly believe that the best recommendations come from your friends, and that’s what we try to focus a lot on at Untappd. That’s an important part that gets overlooked when it comes to ratings of beer, and that’s what we’re all about in terms of discovery.”
When BeerAdvocate was launched, the site allowed its growing user base to review a beer by attribute (look, smell, taste and feel), and give it a rating. A similar philosophy guided RateBeer, a review site founded by Bill Buchanan and now run by executive director Joe Tucker, who discovered the site in 2000.
Tucker, who was developing an interest in wine at that time, had seen that reviews from prolific wine critic Robert Parker could make or break a wine or winery, and found the process inherently flawed. “I knew from my physiological psych background that that was ridiculous because we all have different palates, we all have different aroma sensitivities, those chemical senses that we have are varied in humans and so one person’s taste isn’t another person’s taste, physiologically. So to have one source saying, ‘Yeah, this is good,’ or ‘This is bad,’ is completely preposterous,” says Tucker.
Instead, RateBeer made sense: a community powered by thousands of palates and ratings and tastes. Today the site has 357,000 active members and 6.24 million reviews and wields significant influence.
RELATED: Tasting Beers with Flawless Consumer Ratings
Tucker and RateBeer experienced that influence in 2005 when RateBeer named Westvleteren 12, a quadrupel ale made by the small Trappist brewery at St. Sixtus abbey in Vleteren, Belgium, the best beer in the world. The ranking was picked up by major news outlets like CNN, which drove drinkers to the brewery and drew attention to a small brewery in a quiet area that wasn’t quite prepared for it, says beer writer Tim Webb, co-author of Good Beer Guide Belgium and Pocket Beer Guide.
“This abbey is in a quiet area of West Flanders; there’s little country lanes going to it; the nearest town is about two or three kilometers away. It’s supposed to be a very quiet place where you can just plow the fields and have some contemplation,” says Webb. After the announcement, lines of eager beer hunters formed daily outside the abbey, and the Belgian media reported fistfights in line. Meanwhile, the brewery couldn’t even begin to keep up with the newfound demand.
“That was a good kick in the pants,” says Tucker. “It really got us to think about our math and the way we were delivering our information, the way we were putting together our list, the way we were notifying people about it.” Now, they try to be a little more gentle, Tucker says. He learned to talk to the media to explain what the honors mean and adjusted the math to decrease the volatility of ratings and stabilize the rankings, which are compiled from user reviews.
Unexpected Praise
Arizona Wilderness Brewing Co. stands in a plaza along a busy six-lane road in Gilbert, Arizona. These days, construction crews are working alongside brewers as the 2-year-old brewpub is going through a sooner-than-expected expansion.
Arizona Wilderness Brewing Co. was named best new brewer in the world in 2013 by RateBeer.com. (Photo by Ken Burnett, courtesy Arizona Wilderness)
Jonathan Buford opened the brewery in September 2013 with partners Brett Dettler, the business manager, and brewmaster Patrick Ware. In its first year, Arizona Wilderness brewed about 100 different beers and highlighted ingredients grown by local farmers: herbs, spices and citrus from an urban farm in Gilbert and a heritage grain called Sonoran White Wheat that it uses in Belgian styles and Berliner weisses. That experimentation, frequent turnover of new beers and the fact that the beers were only sold on premise created a buzz locally and drove drinkers to check out what was happening in Gilbert.
Five months after opening, RateBeer released its annual Top New Brewers in the World list. The list, compiled from user ratings of beers, ranked the top 10 highest-rated new breweries. Out of more than 2,600 breweries that opened in 2013, Arizona Wilderness was named the best new brewer in the world.
“We weren’t prepared for what was coming next, which was just a massive amount of response of people wanting to see why we were called this,” says Buford. “Effectively, it catapulted us into a new realm of growth that we had never planned for. It’s like cutting your legs off and learning how to walk. You can’t learn quickly enough.”
It forced them to skip from year one to year five of their business plan in one month to meet demand, he says. They scrambled to keep beers on tap, added a few more 15-barrel fermenters to expand capacity, beefed up their staff and hired an experienced brewpub general manager, Matt McCormack. After nine months, they sought to expand with a new brewery next door, scheduled to open in spring 2015, which will have the capacity to produce about 2,200 barrels annually (they brewed 1,000 barrels in their first year) and includes a bigger kitchen and a humidity- and temperature-controlled barrel-aging and sour beer room.
The Hype Factor
The most difficult piece to nail down numerically or theoretically in beer ratings may be the “hype factor”—the buzz that surrounds a brewery and the expectations drinkers might set for a certain beer or brand. Buford reflects on the pressure to meet demands and the opportunities that it afforded them. “It felt like a bottomless pit we never would get out of,” he says of the crush after the title. “A year later, boy, are we having fun again. We’re being ourselves. [We’re] not so worried what people want from us, think about us, expect from us.”
The title itself, while an honor, has always felt a bit soulless anyway, he says. They never used it in advertising or put it on beer labels or their website. And what Buford has always been most proud of is not that their brewery was named the best in the world, but that they built a brewery in Gilbert. “Really, what I want to contribute to our hype is that we have brought culture to an area that was dead of culture,” says Buford.
Feedback Mechanisms
“The most valuable piece for me is going to be a validated panel backed up with some kind of analytical equipment,” says Matt Brynildson, brewmaster of Firestone Walker Brewing Co. “That’s not what you’re going to find online. That’s a completely different type of feedback mechanism.”
Once a beer leaves the brewery, it often faces variables often out of the brewers’ control: how it’s stored, how it’s poured into a glass and at what temperature, how long a drinker stores a bottle, or whether the beer travels out of its intended footprint in the hands of a beer trader.
Compare that with the Great American Beer Festival or the World Beer Cup, where a brewer ships his beer directly to be tasted blind by a validated panel of judges, typically industry professionals, and only against other beers of the same style.
It’s a similar practice at the Beverage Testing Institute (BTI) in Chicago. There, panelists judge only 25-35 beers in one session to prevent palate fatigue. They taste at the same time every day, usually 10 a.m. to 1 p.m., in a temperature-controlled room at 68 degrees, outfitted for maximum natural light. The beer is chilled overnight and served between 53 and 58 degrees in Riedel stemware. Panelists taste all products blind and do not know any of the brands or prices—just the category they are tasting that day—and scores are translated to a 100-point scale. Rankings from BTI appeared in this magazine for many years.
While beer consumers may just as carefully smell, taste and critique a beer, an online panel is more dissonant than harmonized. That means diverse opinions and a slew of feedback for a brewery to sift through, says Jeremy Danner of Boulevard Brewing Co. As a full-time brewer turned ambassador brewer, Danner works between the brewing and marketing departments, runs much of the brewery’s social media and scans online forums and rating websites. “There’s two mindsets in that breweries can care or they cannot care about what people are saying online. I think it’s about being able to have a critical eye to sift through what reviews or blogs or opinions you really find valuable or influential,” he says.
As an ambassador brewer for Boulevard Brewing Col, Jeremy Danner runs much of the brewery’s social media and scans online forums and rating websites. (Photo by Annie Laura — 611studios.com)
In a perfect world, Danner says, beers would be rated on objective quality and execution of intentions versus personal preference.
“I think the best and most fair way to rate a beer is based on the intentions of the brewer. Our pilsner [KC Pils] is the second-lowest-rated beer on Untappd, and it is a really good beer and 100 percent of what we intended it to be. But because it’s not an exciting or extreme, crazy beer, it suffers in reviews,” says Danner.
Words Can Hurt
More painful than the sting of a negative review is the fallout from rumors and incorrect information about beers, which can be dangerous for a brewery’s reputation. Discrepancies range from the minor—Smith of East End Brewing says that its 3.8% ABV Southern English Nut Brown Ale was listed as 8.0% on a rating site—to the more serious, like claims of a beer being infected.
Cory King, founder of St. Louis’ Side Project Brewing, experienced the latter a year and a half ago. The rumor started when several reviewers on BeerAdvocate posted that they thought Side Project’s The Origin, a hefty blended dark beer aged in rye barrels with vanilla beans, was infected, describing the off-flavor of buttered popcorn, which is indicative of diacetyl. King had the beer tested at nearby Schlafly Beer. The results came back negative for infection, but the damage was done.
“Even though its [sic] not infected, it sucks that all it takes is one person to post it online and now people are questioning it. I can defend myself and my beers on here, but I cannot reply to people reviewing The Origin as infected, and all it takes is one palate to prematurely cry infection to cause a whole stir among the beer community of the other forums,” King wrote on the beer website STL Hops’ forum.
King’s frustration highlights an inherent problem with any online beer review: If it’s not possible to know any reviewer’s taste or palate, how can you trust his or her opinion of a beer?
Jake Austin, founder of Austin Street Brewery in Portland, Maine, says he stays off Untappd for that reason.
“I avoid Untappd [because] it just seems like an easy way to spout off about beer, but you don’t actually know how well-educated they are; you don’t know their palate, what they like for beers,” says Austin. “I think you have to take Untappd with a gigantic grain of salt as far as judging what you want to drink.”
Yet Austin still checks BeerAdvocate and RateBeer reviews about once a week, which he feels provides greater context about the drinker and more in-depth reviews. As a 1-year-old brewery, he wants to make sure his opinion of the beer is matching up with consumers’ thoughts. So far, he says, things have been lining up perfectly.
Heather Vandenengel
Heather Vandenengel talks about beer on the Internet on Twitter at @heathervandy.
Someone with an educated opinion will try a beer and ask “does it taste how it’s supposed to” before reviewing. A novice drinker will rate solely based on how they like it. I see a lot of reviews on Untappd where people will give a beer a low rating and in the comment say it’s because they don’t like the style.
If I try a style of beer that I know is not my favorite, most times I will just check in and not rate. I am however, using Untappd as a tool more so for me than for others so I’m rating them based on how well *I* liked the beer (not on how well it may have been made). I use my ratings to refer back to and remind myself of what I liked and didn’t like. That being said, I will sometimes throw in a “hop profile is not my taste” type comment to indicate that it may be great for others.
I also use Untappd that way, to rate how I liked a beer. That way I can refer back to it. Besides, I am the most important piece, not some BJCP profile that doesn’t mean anything to my palate.
I definitely see great value in your statement, Jason — it’s just plain good to know what you’re reviewing. We at RateBeer encourage experience and associated learning because of this. At the same time we ask reviewers to review “hedonically” and not to style. It’s a valid and alternate way of reviewing beer. While we do have Style scores and ask people to appreciate a beer that is made to a particular style, we’re seeing more and more brewers color outside of the lines — my feeling is that they should be as free as chefs to do so — and we want to allow brewers and reviewers the freedom to think about beer outside of those constraints.
The worst review iv given on a beer was simply “im not a huge fan of this” if i dont like it i dont talk about it. I have never had to drain pour anything being because i have a pretty open mind about “exbeeriments” i like to try and figure out what the brewer was going for. If i cant get it figured i wont write about it. Same thing as a fine dining food critic wont write about Mc Donalds. I also like to grade beer on price, availability and ABV. Some beers are exempt of this grading curve.
Really? You rate beers based on price and availability?! What does that have to do with anything? You are missing the point of reviewing beers, and contributing to the problem in a way that the professional side of the business never even considered!
I sometimes consider price and availability too, but only to help a beer. I would never deduct. For example I bump up SweetWater IPA because it’s available everywhere here and it’s a great bang for the buck.
When considering price, one must define this as value, in other words, is what I’m drinking worth what I’ve paid for it?. In everything we purchase, there are always bargains, fair price, and overpriced items. It’s all about finding the best you can afford, it requires research.
I really don’t pay much attention to these sites other than I want whatever factual info they may offer to be correct. Everything is so subjective – and everyone has such different levels of knowledge and sophistication – that it is virtually impossible to make everyone happy. We just do the best we can to ensure that what the consumer purchases is what we had in mind when we made the beer – and do it consistently. If we do that – and they enjoy the experience and can purchase it repeatedly – then as professional brewers we have done our job and lived up to our brand. It’s not easy.
Beer is meant to be drunk, ideally in a social scenario, not rated to some system (BJCP). Rate the beer to your enjoyment.
Thanks for the article and mention here. The author, Heather Vandenengel, captured the value and pitfalls of what we do very well, and illustrated the distinctions between the online review sites fairly.
I would completely agree with taking online reviews for what they are, online reviews, i.e. I came here for the comments. “They don’t just let anyone post online.”
Not knowing what a reviewer palate might lean towards, I look at all reviews from people I don’t know with a grain of salt. But it is still a good tool for figuring out what an unknown beer might be like for those of us that don’t have the validated panel information readily available.
I also find it amusing that Boulevard Brewing stands by their second to last rated pils as “the people don’t understand the intention of the brewer.” That’s all fine and well to have a beer the way you want it, dull and uninteresting, but at some point you have to sell it. While everyone’s palate may differ, I would argue that, for the most part, the majority of people who would go to the trouble of posting a review would recognize a great beer when they tasted it. I for one don’t like imperial IPAs but I know a good one when I taste it.
And I’ll be the first to say that I might have a skewed perspective based on what, where, and who I drink with but I wouldn’t discount the online world entirely if I were a brewer.
I feel that it’s important to point out that I didn’t say what you quoted me as saying. I didn’t explicitly say “the people don’t understand the intention of the brewer” regarding our pilsner nor did I imply it. My point is that when compared to more extreme, aggressively flavored styles of beer, more subtle, balanced beers tend to suffer.
Sorry Jeremy, I was paraphrasing and should have been more clear about that. I agree that the balanced more traditional beers don’t have the excitement of some of the crazy beers out there (Kölschs and Pilsners especially), but at the same time not all traditional Pilsners are at the bottom of the rating chart.
I’ve been homebrewing for 7 years now and I know the blood, sweat, and tears that go into getting a recipe the way you want it. My point is as a homebrewer, that’s fine, I’m the only one drinking it. But as a commercial brewery who has to sell the beer, it’s not just about what you want anymore. And you guys might be moving your KC Pils, I don’t know. But the question has to be asked what is your potential lost profit by sticking with a low rated beer. Like Untapper or not it has influence on peoples perception, fair or otherwise
Thanks for clearing that up. It was the use of quotation marks that made me feel you were quoting me/Boulevard versus applying an implied theory regarding reviews to our recipe formulation and decision making processes.
I think the reason any brewery sticks with a beer that might not have the best reviews is that the beer isn’t intended for the folks responsible for writing most of the beer reviews. As folks who are so deep into our beer geekery (a very positive, friendly term), we forget that there are beer drinkers out there (me included at times) who just want to sit down and drink a beer that doesn’t require much thought. I think pilsners, wheat beers, golden and amber ales all fall into this category of super approachable beers intended for a more mainstream audience than the 1%.
Our best selling beer, Unfiltered Wheat Beer, has a score of 44 on Ratebeer, a 78 on BeerAdvocate and 3.55 stars on Untappd. This demonstrates that there isn’t necessarily a relationship between sales and review scores.
Neat article! As an avid Untapper and Rate Beer Advocator but also former brewery and distributor employee, my relationships with these sites is a little polarized. I hate seeing “meh”. I am frustrated by the lack of appreciation for some very hard to brew and classic styles and the arbitrary elevation of others that may or may not be as deserving as the hype implies. Still, these sites are great for the community and great for the industry in my opinion. Thank you for the little bit of history and the attempt to look objectively at both the benefits and pitfalls that come with beer social media.
I’m a homebrewer with 21 years of experience and a BJCP certified level judge for the past 5 years. I use Untappd to rate beers because of the convenience of it compared to Ratebeer or Beer Advocate where I also have logins. I have checked in over 700 unique beers on Untappd and I try to offer thoughtful comments although on occasion I must admit I’ve given the proverbial “meh.” I use Untappd primarily to track beer I’ve had and my impressions of them, but the social aspect of it is also somewhat addicting. I do try to think about the breweries when offering my opinions as I understand each beer they put out is a labor of love and attempt to qualify any criticism. However I also feel I’m doing my fellow craft beer lovers a disservice if I don’t comment on perceived flaws in a beer such as DMS, diacetyl, acetaldehyde and oxidation. It’s a fine line. I would encourage those of you that rate beers to be thoughtful about it and use flavor descriptors, point out positives and any perceived negatives, but be respectful in doing so. I apologize for the less than thoughtful reviews I’ve submitted cause the bottom line is I’m a beer lover. I can enjoy any style for what it is including macro lite lagers. My true passion is the culture around craft beer and the people behind it. So drink on, enjoy it and be respectful. Cheers!
“However I also feel I’m doing my fellow craft beer lovers a disservice if I don’t comment on perceived flaws in a beer such as DMS, diacetyl, acetaldehyde and oxidation.”
Keep it up.
This is exactly what I would have written, minus your experience.
To me, Untappd is about the social side: discovery, sharing experiences, making jokes, etc. And yes, I’ll tip my friends off to defects if I find them.
The review system may not be perfect, but nobody’s is: take them all with a grain of salt. At least with friends, you can encourage them to take another chance when they’re “wrong”—on another day, your IPA-loving buddy probably wouldn’t have disliked Furious.
As a professional brewer, I never rate my beers anymore. I know the effort that goes into them. May not be my cup of tea, but someone else may love it.
“had seen that reviews from prolific wine critic Robert Parker could make or break a wine or winery, and found the process inherently flawed”
Ironically this is very much the role RateBeer has today. And while no single expert is causing this, it seems that the RateBeer hype is very much driven by a elitist group of expert raters, favoring a certain group of breweries. It seems more and more like an echo chamber.
As a ratebeer user, I rarely look through the comments before I try a beer. I only look at the comments if I am trying to describe a particular flavor I am unable to put into words.
I believe it is good practive if you’re not an IPA person to rate an IPA for what it is, I try to make note of not being a fan of these beers, but it has not stopped me from rating a couple as top tier beers! Happy drinking, happy rating, probst!