Ratings: Bah, Humbug

This morning in my inbox I got an email from eMusic asking me to "rate" Hank Crawford's album "Tight." Ratings seem to have become a ubiquitous feature of the internet. When I read audio equipment reviews on audioreview.com, I have to be told how many stars the piece got from how many reviewers (i.e., readers). When  I look at pictures on PhotoSig, I have to learn how many people gave it however many "thumbs up" symbols. (The maximum being three. What?) I have to say, the rating culture of the internet is not for me.

As a sometime critic you might not suppose I'd say that. Isn't "rating" things what criticism is all about? The Sony A900 gets five stars. The Sigma DP2 gets two. Done and out.

That's not it, of course. "Rating" anything—especially, taking averages of many peoples' ratings—misses all the subtlety. First of all, whether an album of music gets five stars or two stars doesn't concern me at all. I would hope that I can have great experiences with music that leaves the mass audience cold. Certainly, what turns on the mass audience—broadest spectrum, lowest common denominator—leaves me cold. Mass ratings systems are just looking for common agreement. They're setting community standards, if you will. That might work for refrigerators. Maybe. It doesn't work for art.

I'm looking for an experience with art. An immediate connection between it and me, now. However many of my fellow humans think crasher squirrel is cute or approve of a Celine Dion song, that does not necessarily predict a good experience for me. My taste isn't mass taste. Mass taste isn't for me. (In fact I have a bias the other way—against mass taste—because historically that's been a better-than-neutral indicator for me.) I love discovering music, books, pictures, and so forth that I love, but I have no condition attached that other people must agree with me.

Plus, how can I know when something will hit me? I recently had a fantastic experience with the first three cuts of Duke Ellington's Piano in the Foreground, an album I've owned forever. (The fourth cut, not so much—it tries to make up in length what it lacks in interest.) Thirteen beautiful and brilliant minutes of music that a 15-year-old Kanye West fan would give one star. Me? Last week, three stars. This week, five. No, that does not average to four.

Even when I "rate" cameras, I assume people will disagree with me, and I completely endorse their right to. I go into writing about a camera knowing that somebody out there will love (and, more importantly, do good work with) what I think is the worst piece of glack in the world, and that somebody else will dismiss out of hand a camera that I think is well nigh on the Holy Grail. That fact does not confront me.

It also matters to me who's doing the rating. It gratifies me that when I recommend a photography book as being outstanding (my current ga-ga favorite is this one, which I totally freakin' love and which you might as well bury me with), lots of my readers go buy it. But when I recommend a random non-fiction book on a random topic, like this one, which I found immersively fascinating and a great read, well, pretty much nobody goes and buys it. Why? Because you know my taste and opinions about photography. You give me credit for expertise, and you probably have a sense whether you are going to enjoy my recommendations or not. But you have no such faith and confidence in my general taste about items outside of my expertise, in which case I am just another guy on the internet.

That's fine. You're allowed to be wrong about my general recommendations, which are spectacularly insightful and rock-solid reliable. Just kidding. How'd you like this post? How many stars would you give it on a scale of five? How many thumbs-ups? The maximum is two, if you're a humanoid, one if you lost an arm in battle. You may additionally give it up to two big-toes-up, if you're barefoot.

That Danny Lyon book has totally reordered my opinion of him as an artist. It gets eight stars on a scale of five. And I don't care if 41 raters on netnet.com agree with me, it does.

Mike

Send this post to a friend

Comments

[pb]

5* post

Sounds like me, feels like me. My Being out of main stream thinking and being poor is sometimes a bear, but I ain't changing the Not Bot part.

Thanks Mike.

pb

[Chuck Albertson]

Give the album an 88. It has a good beat, and you can dance to it.

[Dale Moreau]

Hey Mike,

Glad to see your back.

Fun stuff, thanks. Don't rule out events that create lightbulb moments in what may have been simply curious and are now enlightened.(It's happened to me time and time again with Miles Davis. What was once "interesting" is now Remarkable.
Recently I had the opportunity to attend an intimate presentation by Joyce Tenneson at an openening at The Portland Museum of Art. Prviously, Joyce's work had been "interesting". Within a couple minutes of the start of her presentation, her personal work became remarkable and inspiring. Life is a lot of fun if you let it be. Your humbug personality is fun for us out here.

dale

[Alton Marsh]

Couldn't agree more with being skeptical of "ratings," which is my way of saying photo ratings on photo sites are bogus. I think it is a 4.5 but you gave it a 3. So what? And for people to actually let their emotional health ride up and down based on the rankings their photos get is ridiculous. Who cares?

[erlik]

User ratings are oh so three years ago.

They usually serve the same purpose as vox pops on TV: people who have no idea what's goin' on tell others what they think about what's goin' on.

That is, they give an illusion of meaningful participation.

Thus ends my curmudgeonly post for today.

[Marty]

I give this post 2.71 stars, because that Danny Lyon book did turn out to be unpretentious and and revealing of talent...two things 9.146 photo books out of 10 are not
...but the post was entirely too long and verbose...we had the point at "I have to say, the rating culture of the internet is not for me." ...totally summed it up right there dude.
I gotta get back to twittering about facebook on myspace now.

[Dennis]

Mike, I agree wholeheartedly. I was going to disagree about certain things - I'll go to amazon & hope for enough reviews for statistical significance on things like ... appliances :) I mean, if most people are happy with a fridge (I'm honestly shopping for one right now) odds are I will, too. I know what the features are, but don't know if it's noisy or prone to breaking down in 6 months.

But it doesn't work for art, as you say, and it doesn't even work for things like camera lenses. A couple of websites have 1-5 ratings of lenses on sharpness, wide & tele, open & stopped down, but I find that data useless no matter how many scores. I'd much rather read an opinion (or many opinions) and better still, try it out.

OTOH, I'm happy to see that anything I'm looking at has many ratings on amazon or b&h or any other site where the ratings are accompanied by reviews, because then I can drill through the reviews and some of them, people spend a good deal of time on.

The ratings remind me of earlier days of digital cameras when there were significant differences between the Oly prosumer and the KM prosumer and the Nikon prosumer and people would take the time to put together spreadsheets rating the cameras on 20 or 30 different attributes, each with a weighting that supposedly emphasizes the relative importance of each feature (to the person doing the spreadsheet, but usually posed as if it were universal), and the camera with the highest tally would "win". I always found reviews and specs good for narrowing down items for consideration, and then things usually get subjective for me. I sometimes can't even explain my choice; it just "feels right".

But most importantly, if the A900 is a 5 and the DP2 is a 2, how do you rate the E-P1 ?

[Jim Richardson]

Mike,

Recently I was touring iTunes looking for recordings of some of my favorite music when I found someone reviewing a recording of a piece I like. To paraphrase, they said "This is really great recording and the composer should be congratulated." To which I wanted to say, "Well, that would be Tchaikovsky and he's dead, in case you hadn't heard."

So much for unknown (and unknowing) reviewers.

[Barry Margolius]

Well, you got me. I'm generally a lurker on this site, but I so disagree with this posting that I had to respond. Actually, I think most (if not all) of what you said is true; it's just meaningless.

Giving a rating in a review in no way mitigates the rest of the review. Anti-rating bigots seem to think that just because a twenty page review includes the phrase "3 Stars", all their readers will completely ignore the review. In truth, most of the readers will ignore the review, but that was destined from the beginning. I'd bet that almost everybody who would read the review without the stars will read it despite the stars.

I'm admittedly a bit of a dilettante. I like to know a little bit about lots of things. For me, it's fun. There's no way I'm ever going to read an in depth review of a Jazz album/CD - I just don't care enough (sorry Mike), but I would like, as a matter of passing interest, to know if a new Jazz CD is great, or mediocre, or lousy. Once in a great while, I might even read the review of a "great" jazz CD, and maybe even buy it.

The Sony A900 is a great example. I have a strong interest in digital photography, but my interest is not in DLSR type cameras. I do not own a DSLR, and I probably never will. I just don't have that kind of passion any more. But, I am interested in Sony, and how their foray into digital photography is working out. Have they really assimilated the Minolta brand and turned it into a valuable Sony brand? I might read the conclusion paragraph of an A900 review, but I'd certainly take a look at a rating for the A900 if it were available.

The raters always seem to focus on the aficionados, while the ratings are for those of us with mild interest. I don't see how a rating could hurt: it allows the grazers to enjoy the review and does not "damage" to those that want to read the review in depth.

-barry

[Robin Parmar]

Crasher squirrel gets four thumbs up from me.

But then again, they say I'm all thumbs.

[Alex]

Being in market research I deal a lot with ratings and interpreting them. In the end, what I find interesting isn't necessarily what the overall rating of any single thing is, but rather what the people who give it high and low ratings have to say.

We quite often will pose probing questions of the people who strongly like/dislike something or agree/disagree with a statement. In seeing why they would have a strong opinion you can learn quite a bit.

People don't always love or hate something for the same reason; the reason someone hates something could be exactly why you will love it. A rating alone without context is mostly meaningless.

[Keith Loh]

You know what is funny? eBay ratings. I can't tell you how many times I've rated a vendor as average and then had them come back to me and remind me that anything less than perfect affects their standing in eBay. Well then, why even have a 1-5 star range? To me 3 is about okay/acceptable and 5 is perfect and that is how I rate anything with a 1-5 star range.

[yz]

I rate this article five stars out of five...
Anyone else?

[Kurt Peters]

Cameras like cars, guitars, guns are objects. Because they are essentially tools, they can be objectively evaluated in light of their intended purpose.

We can't say the same about art because we are never really sure of its purpose. When it is good there is a transcendence beyond the physicality of the thing itself.

I believe photography is more craft than art. Like writing, and filmmaking, its narrative structure is one of its strengths that is diminished when it presumes itself art.

People think craft is a bad thing, somehow subordinate to art. I would disagree. The lack of craft is why we have so many ugly tables and boring photographs.

I'm sure everyone can build something that functions as a table, just as everyone can snap a photograph. That doesn't mean I have to like it because I like photographs or setting things down on tables.

I wish I still had the URL but someone put Henri Carter-Bresson's photo of the bicyclist, Hyéres, France 1932, up for critique on Flickr. The "helpful" comments were astounding. I've not heard of photoSIG before this article but the site is a repository of photographic kitsch and cliché.

The fact that most people couldn't recognize a good photograph from a bad one is troubling but I wouldn't put it down to just personal taste. It's a cultural conditioning that is hard to break through.

[Jimmy W.]

I read this post in Google Reader. And there's a box I can click to say "I like this post". Even this blog post is being rated.

[Paul McEvoy]

Mike,

just for the record, what was your opinion of Danny Lyon before you saw that book? Before your opinion was reordered? I always thought Bikeriders was one of the best photography books ever, was a little sad that there wasn't much more of his work available, but he was always in the pantheon of amazing photographers. Was your opinion of him changed significantly by the new book? Or did it just remind you of him? I'd put him in with Capa and Cartier Bresson. I'd give them all 5 shutter clicks. Or 5 rolls of film. Or whatever you like. That's out of 4.

[K.Kafka]

The late, great Hank Crawford ?
Why would eMusic ask you to review an album that was recorded and released in August 2000 ? Are they trying to push sales for this 9 year old product ?

[mike in montreal]

Yes, crasher squirrel is plenty cute but that wasn't its significance for me.

The picture says a lot about authorship, technology, the role of the happy accident - plus it's just plain funny, cute or not.

You've run similar stories before. I didn't expect you to be so dismissive.

I give your reaction a one-and-a-half.

[Philip Jelatis]

An individual's opinion is a filter. If you learn what a certain filter is like, then with time their opinion carries weight. Ratings like you discuss, or Top-40 music for another example, are much more about commerce and popularity. We don't know the raters. They're diluters of opinion. High sales and amounts of clicks or thumbs are confused with excellence. Which is why if a film goes straight to video, there's a good chance I might like it.

Have you any idea what happens to your off-topic recommends? I bought and read "The Pencil", gave another away as a gift, and told half-a dozen people. I easily turned on at least 20 people to the song "Hide and Seek" and to the linked artists and the original artist Imogen Heep. My kids checked out The Blakes. Sometimes your readers can even return the favor! Here's something musical and off-topic which would not surprise me to have discovered in one of your posts. Did I?
http://videos.komando.com/2009/06/18/african-thunderstorm/

[Tim]

4/5 for having a rant to which I can relate.

-1 for gratuitously having to reference that cheeky squiggle in order to appear relevant.

Not entirely unrelatedly... Over here, we right-pondians have BBC Radio 3, a station that broadcasts music, criticism, evaluation (saturday morning _CD Review_), evensong services from cathedrals on a sunday afternoon, first-rate performances, the lot.
We also suffer from "Classic FM", which broadcasts noise (second-rate), where the presenters (third-rate) refer to movements as "bits", which thinks a week of Concierto d'Aranjuez every morning should be followed by a month of Borodin every morning, and, where others would celebrate Easter by broadcasting a performance of one or other Bach Passion, Classless FM runs a "chart" for the occasion, just like they do every other sodding week in the year for any other nameless cause.

Photo.net has/used to have a two-axis rating system for execution and originality, or something. That probably doesn't span all the aspects one could consider, but it's better than just the one dimension.

ObIntelligentStatement: this is why I want a diploma course in *reading* a photograph, how to criticize a work of art, in order than I might make better photos worthy of pondering myself. Criticism should be reading, not rating.

[Michael Leuchtenburg]

The thing ratings are the most useful for is recommendation systems. Have you ever used Netflix or Amazon? You know the recommendations they give? They're based on ratings. Also on rentals and purchases, but ratings are a big part.

[Robert Noble]

Is "rating" stuff largely a "guy" thing? Does the prevalence of "ratings" on the Web reflect the prevalence of men spending a lot of time online?

For that matter, has a woman ever written a comment here in TOP?

[Mike Johnston]

"Why would eMusic ask you to review an album that was recorded and released in August 2000?"

Because I bought it recently. Oh, and they didn't ask me to review it, they asked me to *rate* it.

Quote:

"Rate your music, Mike! Recently you downloaded 'Tight' by Hank Crawford. Tell us how much you like it by clicking on the stars on the album page."

Mike

[Mike Johnston]

"For that matter, has a woman ever written a comment here in TOP?"

Of course! Many times.

Mike

[Mike Johnston]

Hi barry,
Thanks for your good (first) comment. I can't argue with what you say, except to note that you're assuming that a review goes along with the rating, which isn't always the case.

Mike

[john robison]

Ratings may not mean much but reviews of products and services can save a lot of grief and money. Case in point. I live in an apartment building where the only air conditioner I can use is a portable roll around unit, conventional window units can't be installed. Before spending the $400~600 on such an appliance I checked peoples opinions and experiences with them. Almost all reviews, of any brand, caused me to conclude that they were inherently inefficient and poor performers that took an enormous amount of power to cool even small areas. That information saved me dollars and dissatisfaction.

[K.Kafka]

Mike,
Thanks for the clarification.

[Mark Probst]

Yes, ratings as averages are close to useless. Where ratings do make sense, though, is when correlations are between raters are computed. For instance, suppose there's a guy somewhere with whom you have 20 rated songs in common (i.e. you both rated those 20 songs), and both of you rated each of those songs almost identically. What do you think will be the chances that you'll like a song he rated highly that you haven't heard of yet, or vice versa?

Amazon, for example, does something like this.

[Dale Moreau]

Kurt,
With well executed craft the best one can expect is that the total achieves the sum of its parts. With "Art" the sum is exceeded.

[robert p]

@Dennis - go look at LIEBHERR fridges if you can. We just got one. These guys make port cranes and mine trucks, they seem to know what they are doing with making stuff that works.

@Barry - try "The Black Saint and the Sinner Lady" by Charles Mingus

@Mike - ratings - yeah these star things are hard. I am currently trying to buy some speakers (a subject I know you enjoy) and the price/size/quality is ending up looking at either B&W 686 or TANNOY DC4, and there is nowhere I can find anyone real who has heard the TANNOYs, but there are lots of people who have heard the B&Ws.... "Buy the B&W" they say... yet HI-FI Wolrd says that the Tannoys get 4/5 stars and the B&Ws 3/5......

So what do I believe..... and if I can hear the B&Ws in a store, but can't hear the Tannoys before buying, would I be a fool to believe an extra star and buy them...?

Sheesh.

Liebherr fridges though. They work.

[Clayton Lofgren]

Ratings are very much a girl thing, but the only thing they rate are guys.

[MESullivan]

Kurt, I believe this is the url you needed.
http://www.flickr.com/photos/andrerabelo/70458366

To me it doesn't matter if it's art, craft, or object. You have to know a little about the reviewer to know if their review is actually worth entering into your decision making.

I remember seeing a post by someone at Miranda.Com who had posted a lot about different cameras and lens in which he had been amazed to learn you could sharpen photos in post production. He learned this of course after rating many lens and bodies.

I've seen the same in reviewers you would think would know better in that long after they had reviewed a piece of equipment they found out something about it that was in it's basic operation. Did they even use that piece of equipment they reviewed??

I am far from an expert in anything photographic but then I don't pretend to be as so many who review gear and photos made with gear.

[MESullivan]

I almost forgot, I will always rate this a 11 on a 1 to 10 scale.
http://theonlinephotographer.blogspot.com/2006/06/great-photographers-on-internet.html

[David Dyer-Bennet]

The question is, what do they do with the rating? If they've got your personal response to a lot of things, and other people's response to a lot of things that overlap you, and then one of those people is looking at something you've rated -- they can do a LOT more than just average the ratings. That's what Amazon is doing, for sure, with the ratings you give them. Don't know about other places. (Remember Alexandria Digital Literature?)

[Timo]

How did I like your post?

1. Wow, that "Hide and Seek" is quite a track. Adds a full star to this somewhat tepid rant -- and so we see even a grumpola post sometimes engenders a good discussion. For me, it's like the pleasure of a lucky photo. (But not of a squirrel, of course.)

2. "My taste isn't mass taste. Mass taste isn't for me. (In fact I have a bias the other way—against mass taste—because historically that's been a better-than-neutral indicator for me.)"

Now you're talkin'. Taste is such a fascinating subject, and obviously the elephant in the room in any discussion of aesthetics.

[can of worms] It's not an easy or casual book, but you might really enjoy Pierre Bourdieu's Distinction: A Social Critique of the Judgement of Taste -- nary a mention of ratings, but plenty of discussion of the position and meaning of supposedly disinterested aesthetic judgments in art. [/can of worms]

[Geoffrey Meyer-van Voorthuijsen]

I've been meaning to get that "random non-fiction book on a random topic" ever since I watched this.

[Paul Amyes]

I think with many of the ratings systems that are used people get confused with whether they identify with the item and whether it does what its supposed to do well. My bug bear is camera reviews where the reviewer early on says I use brand X of camera and review camera Y does not use the same control layout so I'll mark it 2/5 for handling. One of the worst reviews of a camera I've ever read was on photo.net. The review was of a Pentax 645, but you'd wouldn't really know that as it was really about the reviewers love affair with his Rollei.

With regard to more subjective things like photos, paintings, music or books how I'd rate things would ultimately depend on my emotional response to it at the time. Consequently this can mean my ratings would then change. These means of expression are capable of eliciting an emotional response and our reaction would depend upon how in tune that emotion is with our general frame of mind at that time.

[James McDermott]

Personally, I give my Pentax K20D a JSR (Jim's subjective rating) of 132.5475 (RAW output) and 119.1572 (JPEG), which compares well to the best bowel movement I ever achieved (a mere 85.3222 JSR), or finest bucket o'poultry-type product I ever ate (14.2). If we don't rate, how can we know if we're really enjoying stuff?

[Paul]

Ah, the trials and tribulations of people who want to have their choices validated. Did I buy the right camera, car or appliance? It's like a friend of mine who buys his cars after reading consumer reports, and buys what they suggest - he has a car that rates well, but is it enjoyable? Maybe they'll add a rating for that. I think we all want to make the 'right' choice,' unfortunately we live in a world with too many choices. Maybe we just want a stranger to tell us we did good.

Personally, I prefer a reviewer who can bare his biases, and tell me what he thinks. And who trusts me enough to make up my own mind, based on my biases and preferences. Just my 2 cents.

[David Bennett]

Mike,
You make a point of addressing your readers as one individual - with some skills but not the last word in skill - in a group of photography aficionados.

You also recommend books and cameras, and as I have an opinion on the subject of ratings, recommendations, and the group - here is the link:

The power of recommendations: looking for the group

[John]

I agree with most of your points. Arts and written works can't be evaluated properly through rating systems. At best they serve as a way for people who don't want to read or write a review a way to participate in the process.

On retail websites, though, I think ratings do serve a useful purpose because they serve as a shorthand for whether customers are generally satisfied with something. If I see a low average rating, I usually dig into the comments to see what people are angry about. Sometimes users will pick up on a flaw that wasn't given proper emphasis in published reviews.

So 4/5 stars. :-)

[David A. Goldfarb]

I've been making this argument for years. Ratings are the downfall of any kind of internet community, in part because rating systems attract people who like rating systems, and there are many such people who prefer reductionism to thought.

So say you have a combination image critique and discussion site. People who prefer reductionism to thought are attracted to the image popularity contest, and slowly become involved in the discussion forum. Not enjoying thought, they don't contribute much of value to the conversation, and the discussion forum goes downhill, and the core community migrates elsewhere.

[Carl Dahlke]

Sometimes the distribution of ratings can be interesting. A CD on Amazon with lots of both good and bad ratings actually inspires me to dig into the reviews more than a set of ratings that are uniformly good. I also use ratings to decide which reviews to look at - I often go for the 1 and 2 star reviews first - especially with tools and appliances.

But the ratings are just teasers for the reviews. Without the reviews the ratings are literally meaningless. 4 stars doesn't actually mean anything, but the words in the reviews can mean something.

[Ludovic]

*****

(But that squirrel still has more going for it than 99.97% of the images uploaded on Flickr last year :))

[JonA]

What I think is funny about the squirrel picture is not that the squirrel is anthropomorphic but that cameras have become so smart that it adjusted focus to put the squirrel in focus and the people and scenery out of focus. You can have a camera that you don't even need to focus or tell it what type of picture you're taking before hitting the timer button? Hilarious! Who knows what picture these people might have ended up with even if the squirrel wasn't there? Hey dummy, I don't care what you're intending to photograph; this squirrel is better than your photo idea! I think it would have been funnier if the squirrel and scenery were in focus, then it would be like the squirrel was on vacation too and taking a picture to remember his trip.

[Nic Mainferme]

Hi Mike!

I looked up Piano in the Foreground and whadyaknow? they have it in MP3. Don't know if that helps you for posting a TOP link for a few pennies or not... http://www.amazon.com/Piano-Foreground-Duke-Ellington/dp/B0002J58OM

I'm listening to the sampler and it sounds gorgeous! What a grand composer and musician... thanks for letting me in on such a great collection!

Nic