Testing, like shopping, can waylay the unwary indefinitely. Beware.
Never being able to stop shopping is one of the downsides of my job. I did stop testing for a while, but only because it was starting to make me crazy. When weighing poorer job performance on the one hand against insanity on the other, I opted in favor of sanity. It might have been a questionable call, I don't know.
Testing is hard to do these days. Electronic sensor-based cameras are incredibly complicated little devices. They can do lots and lots of things. It's been a long time since I thoroughly understood all the capabilities of one camera—I envy people like Thom Hogan and Rico Pfirstinger. I think Thom exhaustively understands the functionality of about 40 different cameras, which, if it's true, puts him 40 ahead of me.
There are three aspects of testing your own equipment that are important. First, you have to design the experiments properly. That takes creativity as well as knowledge. Experimental design is one of Ctein's great talents; his training in physics and his lifelong enthusiasm for that field give him the ability to design tests that actually test the thing he thinks needs testing. Many people start off trying to test one thing, inadvertently test another thing, and never know the difference.
The second important thing is technique. For instance, yesterday I did some quick and dirty (Q&D) tests comparing my X-T1 with my friend's X-Pro1. The results alarmed me, so this morning I did some tests using better technique.
I used a tripod, took care to compare that the settings of both cameras were identical, and bracketed focus. The results this time weren't alarming.
But as you can see, my technique was still not very good, because the exposure isn't matched. That makes visual comparison more difficult. It does not mean the X-Pro1 has better shadow detail and the X-T1 has better highlight detail.
Rats. You remember the pertinent line from Robbie Burns: "The best laid schemes of Mikes and men / Often go awry."*
The third important thing is that you have to be careful to ascribe the test results to the correct cause. For example, fifty years ago hobbyists used to test lenses by tacking a spread from a newspaper on the wall and photographing it. They would then make pronouncements about the performance of their lens. What they seldom realized was that a) they were testing their lenses only at a close focusing distance, and lenses (especially in those days) performed differently at different distances; and b) they were also stressing the flat-field performance of their lenses. Curvature of field might make the corners of a sheet of newspaper look smeary, but not have the same effect on a complex three-dimensional subject.
Another common testing failure is slight misfocus, which torpedoes many amateur testers' efforts right from the start. It's easy to focus a camera approximately, and, it turns out, very difficult to focus it with extreme precision. That's why devices such as FocusTune can be so informative and helpful.
The global conclusion is, camera testers should never extrapolate. You've only tested exactly what you've tested; you shouldn't draw broad conclusions from limited results.
I want to know as much about my equipment as I need to know. My stance toward testing my own equipment is to use it, and observe the results carefully; only if something seems amiss do I perform tests designed to ferret out the cause. In other words, only if I need to know more about it do I learn more about it.
A certain amount of testing when you receive new equipment is self-protective—you want to know quickly that you haven't gotten a defective sample. Pros will often test new equipment extensively when it's brand new, because they need to figure out quickly how it works and what its strengths and weaknesses are.
Among amateurs, some people enjoy testing to such a great extent that it's all they ever do. That's okay for them. My feeling about it, though, is that life's too short. If everything seems to be okay, why go looking for trouble? I see too much when I look at pictures already. Any problems that exist aren't going to escape my notice for long, whether I want to see them or not. It's a bit of a curse.
*Oh, okay, that's actually the English translation. The verbatim quote, I'm pretty sure, is "The best-laid schemes o' Mikes an' Men, / Gang aft a-gley." It could be I've included a slight variation, in the interest of improving the quotation's truth value. Who knows how carefully mice make plans, anyway?
Original contents copyright 2015 by Michael C. Johnston and/or the bylined author. All Rights Reserved. Links in this post may be to our affiliates; sales through affiliate links may benefit this site.
(To see all the comments, click on the "Comments" link below.)
Featured Comments from:
Jim Richardson: "Quite right about accurate focusing and lens quality evaluation. It extends, also, to evaluation of the total system. That is, if you can't get the lens in focus during routine work situations, it doesn't matter how good it is during a test under perfect conditions.
"For example, last year I tested out a medium-format camera (very expensive one with many, many megapixels) and determined that I could not get it precisely in focus in the kind of conditions I was going to face. (Handheld in flooded rice paddies.) My testing made me conclude that I could get better quality from my D800 with the Sigma 35mm ƒ/1.4 than I could with the medium format monster for the simple reason that, in the field in real working conditions, I could get the Nikon in focus more consistently than the medium format. And for me, in my testing, it didn't matter if somebody else (someone more adept than me) could make the medium format work because they weren't going to be there with me taking the pictures. That's the nature of my testing."
Keith B: "When I get new lens, I run out and shoot pictures with it. If in the first 80 or 100 or 150 shots, I see what might be a problem, then I shoot 'tests' using live view focus and tripod. The established, [now] Internet-based testing outfits (DXO, LensTip, etc.) only test one sample of a product. Trouble right there...samples vary. Only Mr. Cicala of LensRentals has found a way out of the single-sample problem."
Gordon Lewis: "I'm not much into lens testing. I seldom get a lemon, and when I do, it's obvious even from looking at an actual photograph. What I do test for though is the camera's operating consistency. Some cameras I've used can't seem to manage consistent autofocus, autoexposure, and white balance, even when aimed at the same subject, with the same framing and distance. These issues don't take long to surface, and if they do, I bid said camera a quick farewell. The only thing I like less than testing is unpredictable automation."
Michael Perini: "There is enough copy-to-copy variation as well as camera body-to-body variation that we test every new lens and every rented lens before we use it on a job. The majority of lenses lately have been good. We test for front or back focus, carefully, and dial in as necessary, and then shoot about 50 pictures including alternating pictures at two different common distances and with a subject walking towards and away from the camera. We look at pictures to screen for acceptable sharpness. We also do quick brick wall or clapboard siding to look for any exceptional distortion.
"So what we do is not really lens testing per se, but more practical system testing to guard against any real negative surprises.
"Mike is right about real testing being easy to do wrong, and flat-field tests showing defects that will rarely appear in pictures. We have uncovered two lenses over the last 5–7 years that really did have issues serious enough that internal factory readjustment was necessary. So finding those early was a big benefit. None of this provides any data about 'my lens is sharper than your lens.' But it does provide confidence that the system can produce excellent results if the photographer doesn't screw them up."
Thom Hogan: "Re 'Exhaustively understands.'
"Well you got that right, it exhausts me. Worse still, it's confusing. For the first time in a long time I'm about to take a long trip with cameras that I can say I've been using for awhile now (okay, one's a new version of an older model, but there's not a terrible lot to learn about it if I ignore the video part ;~). Put a different way: I'm about to take a trip where I can solely concentrate on photography.
"Oh wait, I'm bringing two new lenses ;~).
"We're at a point in photography where we're picking nits. There isn't a camera on the market that won't take perfectly fine photos if you let it...uh, I mean if you control it.
"That said, what's been said by a few others I'll repeat: one of the reasons why I got into testing my gear extensively before using it in production was knowing what I'm dealing with. I believe that photography is really the collection of optimal data, which shows my Ansel Adams learning. I've found plenty of miscentered lenses, buggy firmware, features to avoid, sensor issues that impact my post processing, etc. I prefer to know about those before a moment comes along that won't happen again.
"I'll give a recent example of frustration. I was shooting 4K video in Botswana this summer with a brand new camera (new to the market, new to me) when that leopard started fishing for catfish in the mud puddles as the Savuti dried up. I can't tell you how many times I hit the wrong button (out of several dozen on that camera) while trying to get decent footage of that unique leopard.
"Fortunately, the leopard was cooperative and just kept repeating her actions over and over. But had there only been one big catfish in the puddle, there's a chance that my button dysfunction might have screwed up the shot.
"One correction to one of the comments: mirrorless is not necessarily self-calibrating for focus. Olympus has AF Fine Tune built into the E-M1 for a reason: if you rely solely on phase detect AF (or Panasonic's DFD) to achieve focus, you need the ability to tune camera and lens together. If the camera also uses contrast detect to tweak the phase detect focus point each time, you're going to slow down the focusing system, especially on continuous AF."