Reader Graydon, in the comments yesterday:
How the camera interacts with the raw converter and other software you're already used to matters, and it is, up to a point, an issue with the camera design. A camera with utterly splendid image characteristics so long as it is only used with the proprietary converter shipped with it is a different camera from one with equivalently splendid image characteristics when used with any old image converter. (Presumably the makers of the second camera are much more open with their raw image specifications.) So I think Mike is right to test cameras with whatever raw converter he is in the existing habit of using; where the camera is on the spectrum between 'works only with the proprietary software' and 'works with anything' is an important thing to know. Since the converter he's using is the one with the lion's share of the market, especially so.
Graydon has an excellent point, but it does bring up an issue for reviewers (and, for readers, an issue with reading reviews): how much are we reviewers justified in only using a test camera like we usually use cameras?
My habitual stance is that I can only use test cameras like I normally use my own, and then report on that. For instance, I don't use flash. I have a vague idea that people tend to like Nikon's flash implementation better than Canon's, but beyond that I have no idea which camera is better for flash users. So, when I "test" (I prefer the word "try") a camera, how much obligation do I have to check out and report on the flash functions?
Ideally, a "real" (i.e. complete, publishable) review would speak to all the ways everyone uses a camera, including flash. But that brings up another problem with reviews: to what extent do you want a reviewer to be "faking it" when he pontificates on a specific topic? What I mean is, I never use on-camera flash with any camera, therefore I have neither a "deep" knowledge of the technical and operation issues involved with on-camera flash nor a breadth of experience with different flash systems and implementations. I don't have well-formed opinions about flash. So which would you prefer: would you like me to take a dozen pictures with the flash on and bullshit my way through a couple of paragraphs about how it works, or would you rather just have me tell you up front that I don't know anything about flash and leave it at that?
My preference, as a review reader, is for the latter. What it tells me is just that I need to go somewhere else for the missing information, that's all. That strikes me as both honest and reasonable. If you read one of my reviews and then assume you'd better head over to Strobist for David Hobby's views on a camera's flash functionality, what's wrong with that? David's forgotten more than I ever knew about in-camera and on-camera flash. Who would you rather listen to on that score? I'd rather listen to David, myself.
Iridient, nacreous, shimmering, glittering, sparkling
What I'd probably do if I took the issue of raw converters seriously would be to maybe pick three and quietly begin using them. I'd wait until I had gained some experience with them all, till I felt I had a solid handle on their similarities and differences and how they worked in a variety of situations, and then I'd begin introducing them into my reviews.
Here's what I wouldn't do: I wouldn't use ACR with everything, all the time, and then publish a review of a specific camera C, and then have people complain that C works a lot better with raw converter XYZ, and then go download a copy of XYZ and try it (only with C files) and then start talking about it like I knew what I was talking about. That's disreputable, in my view.
And, as several people have pointed out in the comments, it gives rise to another problem, which is that you can never please everybody anyway, and successive layers of technical niceties only give rise to further levels of complaint. That is, if I were to write a review which said, in effect, "I did all my tests with ACR but anecdotal evidence on the forums suggested that XYZ is better, so I tried that," then I'd get 19 people expounding about what an idiot Johnston is because doesn't he understand that XYZ is a flibbertegibbet app and he needs to use a whapdoodle app such as the raw converter LMNO; at which point the conversation explodes into a flibbertegibbet vs. whapdoodle controversy, with XYZ vs. LMNO overtones.
Believe me, I know. I've been doing this for a while.*
And even then we haven't really gotten anywhere. Because even if I try the C with LMNO, there will be a guy somewhere who has devoted his free time for a year to comparing all the raw converters extant with C cameras, and he'll be absolutely convinced that the best way to convert raw files is to start with C's bundled coverter, save to RIFF, apply the Magic Dust plugin, spin around three times, open the image in Picture Whizz and shout at the monitor, then use his proprietary Action sharpening filters which he will sell you for $9.
What's the solution? Honesty, folks, just honesty. I've reviewed a lot of lenses over the years, for instance, but I'm almost always careful to remind people that for my own work I've always used lenses between 28mm (or equivalent) and 90mm (or equivalent), with 24mm and 105mm (and equivalent) outliers. Within those boundaries I have a lot of experience and a lot of knowledge. So if you come across me writing about a 12mm or a 400mm, you should realize that I don't have a deep basis of comparison. Because I don't use those focal lengths. And how will you know? Because I'll tell you. Simple honesty can overcome any deficiency. Just don't pretend you know more than you know and you'll be fine. That's my turtle-shell, and I'm stayin' in here.
The bottom line: I use ACR and Photoshop. If a camera doesn't perform well with those tools, then it won't perform well for me. It's not the end of the world. The internet is broad and wide, and there are other people out there also writing about the same cameras I write about—or so I hear.
(On the other hand, after some on-the-fly online research, I have just downloaded a trial version of Raw Developer 1.8.2 from Iridient Digital. So in the future you might or might not be reading a comparison between that and ACR. I'd promise to work on it, but, as I'm fond of saying, I have enough things to promise to work on.)
*The 19 people in question would all be veterans of the flibbertegibbet vs. whapdoodle flames on various forums, so they'd be properly radicalized and polarized on the issue, and all their arguments would be fully developed and, probably, long. I call these "imported arguments" when they show up in our comments section, where I usually cheerfully zap them. As with many such blogosphere-related issues, I've developed a sixth sense for knowing what I'm looking at when I see something like this.
Featured Comment by Ken Tanaka: "Just to slightly garnish the dish you've already cooked...I, too, am very much of the mindset that if a camera's files can't be well-processed by Lightroom/Photoshop/ACR (my main platform) it's not my problem.
"During a period of idle curiosity and insecurity a cople of years ago I ended up licensing and testing a basket-full of these independent raw converters (examples: Capture One, Raw Developer, Bibble, LighZone, et al.). I—really—wanted to find advantages over ACR (of that day) that could consistently justify using these products. But while I could find some occasional benefits on certain types of image files I could find nothing that would warrant the inconvenience of permanently implanting any of these products into my processing and cataloging flow.
"I would reach that same conclusion today, perhaps more than ever. Adobe's Camera Raw facility, both as an extension to CS4 and as part of Lightroom, can be adapted to painlessly custom-process raw files from specific cameras through its new profiling facility. But more significantly it has been given the tweak-tools with which one can achieve virtually any result desired. Blaming ACR for conversion disappointment is very, very hard to support today.
"Of course there are, and will continue to be, enthusiasts who swear that some other raw converter is giving them results 'far superior' to ACR. Rather than pointlessly argue with them I simply tip my hat and wish them well. If they've found a digital elixir that they believe makes their images sing sweeter, more power to them. Perhaps they have. But from my seat, perhaps a seat shared by the majority, Adobe has produced stunning gains in its raw image processing facilities over the past years. (I've actually begun re-processing some of old raw images with the newer ACR to great advantage.) That's why it's my horse and I'm continuing to ride 'im!"
Featured Comment by Dale Moreau: "Okay, I was willing to let it go, but since you brought it back up...I do not think that it is unreasonable to expect that a manufacturer's raw converter would provide results that are superior to a 'generic' application. I would think that it would be very reasonable that a specific camera's dedicated software designed by the same team that designed the in camera processing would be able to optimize that same data to output.
"With that said, an argument between generic converters is futile and, of course, simply sticking to the one that serves your workflow is consistent with what one would expect any professional to do. So when one is 'trying' cameras and not suggesting a bench test outcome but simply providing personal observations that suit one's personal style, I see no reason to introduce unnecessary complexity into an otherwise very enjoyable review."
Featured Comment by Robert Noble: "We old-timers can't help but think of the age-old arguments about how photos should be processed. The technical words used in the discussions may have changed, but the tenor of the arguments has not.
"See, for example, this 1999 thread from photo.net on processing photos or this 1999 thread from the LF.info site. (I would quote similar debates from the 1940s and 1950s, but they're not online!)
"SUMMARY: There seem to be two main camps when it comes to processing photos: Those who seem to enjoy the never-ending quest for the magic formula (and who champion the superiority of whatever method they've decided on for now) vs. those who say 'It's not a big deal. Just pick a reasonable, mainstream solution and go with it, learning its limitations and capabilities and changing it only if you're convinced it's lacking.'
"In a sense, both camps play an important role (promoting innovation vs. keeping perspective, respectively). Which philosophy is right for each photographer is a matter of personal disposition."