An Attitude Toward Testing

Testing, like shopping, can waylay the unwary indefinitely. Beware.

Never being able to stop shopping is one of the downsides of my job. I did stop testing for a while, but only because it was starting to make me crazy. When weighing poorer job performance on the one hand against insanity on the other, I opted in favor of sanity. It might have been a questionable call, I don't know.

Testing is hard to do these days. Electronic sensor-based cameras are incredibly complicated little devices. They can do lots and lots of things. It's been a long time since I thoroughly understood all the capabilities of one camera—I envy people like Thom Hogan and Rico Pfirstinger. I think Thom exhaustively understands the functionality of about 40 different cameras, which, if it's true, puts him 40 ahead of me.

There are three aspects of testing your own equipment that are important. First, you have to design the experiments properly. That takes creativity as well as knowledge. Experimental design is one of Ctein's great talents; his training in physics and his lifelong enthusiasm for that field give him the ability to design tests that actually test the thing he thinks needs testing. Many people start off trying to test one thing, inadvertently test another thing, and never know the difference.

The second important thing is technique. For instance, yesterday I did some quick and dirty (Q&D) tests comparing my X-T1 with my friend's X-Pro1. The results alarmed me, so this morning I did some tests using better technique.

I used a tripod, took care to compare that the settings of both cameras were identical, and bracketed focus. The results this time weren't alarming.

But as you can see, my technique was still not very good, because the exposure isn't matched. That makes visual comparison more difficult. It does not mean the X-Pro1 has better shadow detail and the X-T1 has better highlight detail.

Rats. You remember the pertinent line from Robbie Burns: "The best laid schemes of Mikes and men / Often go awry."*

The third important thing is that you have to be careful to ascribe the test results to the correct cause. For example, fifty years ago hobbyists used to test lenses by tacking a spread from a newspaper on the wall and photographing it. They would then make pronouncements about the performance of their lens. What they seldom realized was that a) they were testing their lenses only at a close focusing distance, and lenses (especially in those days) performed differently at different distances; and b) they were also stressing the flat-field performance of their lenses. Curvature of field might make the corners of a sheet of newspaper look smeary, but not have the same effect on a complex three-dimensional subject.

Another common testing failure is slight misfocus, which torpedoes many amateur testers' efforts right from the start. It's easy to focus a camera approximately, and, it turns out, very difficult to focus it with extreme precision. That's why devices such as FocusTune can be so informative and helpful.

The global conclusion is, camera testers should never extrapolate. You've only tested exactly what you've tested; you shouldn't draw broad conclusions from limited results.

Curse
I want to know as much about my equipment as I need to know. My stance toward testing my own equipment is to use it, and observe the results carefully; only if something seems amiss do I perform tests designed to ferret out the cause. In other words, only if I need to know more about it do I learn more about it.

A certain amount of testing when you receive new equipment is self-protective—you want to know quickly that you haven't gotten a defective sample. Pros will often test new equipment extensively when it's brand new, because they need to figure out quickly how it works and what its strengths and weaknesses are.

Among amateurs, some people enjoy testing to such a great extent that it's all they ever do. That's okay for them. My feeling about it, though, is that life's too short. If everything seems to be okay, why go looking for trouble? I see too much when I look at pictures already. Any problems that exist aren't going to escape my notice for long, whether I want to see them or not. It's a bit of a curse.

Mike

*Oh, okay, that's actually the English translation. The verbatim quote, I'm pretty sure, is "The best-laid schemes o' Mikes an' Men, / Gang aft a-gley." It could be I've included a slight variation, in the interest of improving the quotation's truth value. Who knows how carefully mice make plans, anyway?

Original contents copyright 2015 by Michael C. Johnston and/or the bylined author. All Rights Reserved. Links in this post may be to our affiliates; sales through affiliate links may benefit this site.

TOP's links!

(To see all the comments, click on the "Comments" link below.)
Featured Comments from:

Jim Richardson: "Quite right about accurate focusing and lens quality evaluation. It extends, also, to evaluation of the total system. That is, if you can't get the lens in focus during routine work situations, it doesn't matter how good it is during a test under perfect conditions.

"For example, last year I tested out a medium-format camera (very expensive one with many, many megapixels) and determined that I could not get it precisely in focus in the kind of conditions I was going to face. (Handheld in flooded rice paddies.) My testing made me conclude that I could get better quality from my D800 with the Sigma 35mm ƒ/1.4 than I could with the medium format monster for the simple reason that, in the field in real working conditions, I could get the Nikon in focus more consistently than the medium format. And for me, in my testing, it didn't matter if somebody else (someone more adept than me) could make the medium format work because they weren't going to be there with me taking the pictures. That's the nature of my testing."

Keith B: "When I get new lens, I run out and shoot pictures with it. If in the first 80 or 100 or 150 shots, I see what might be a problem, then I shoot 'tests' using live view focus and tripod. The established, [now] Internet-based testing outfits (DXO, LensTip, etc.) only test one sample of a product. Trouble right there...samples vary. Only Mr. Cicala of LensRentals has found a way out of the single-sample problem."

Gordon Lewis: "I'm not much into lens testing. I seldom get a lemon, and when I do, it's obvious even from looking at an actual photograph. What I do test for though is the camera's operating consistency. Some cameras I've used can't seem to manage consistent autofocus, autoexposure, and white balance, even when aimed at the same subject, with the same framing and distance. These issues don't take long to surface, and if they do, I bid said camera a quick farewell. The only thing I like less than testing is unpredictable automation."

Michael Perini: "There is enough copy-to-copy variation as well as camera body-to-body variation that we test every new lens and every rented lens before we use it on a job. The majority of lenses lately have been good. We test for front or back focus, carefully, and dial in as necessary, and then shoot about 50 pictures including alternating pictures at two different common distances and with a subject walking towards and away from the camera. We look at pictures to screen for acceptable sharpness. We also do quick brick wall or clapboard siding to look for any exceptional distortion.

"So what we do is not really lens testing per se, but more practical system testing to guard against any real negative surprises.

"Mike is right about real testing being easy to do wrong, and flat-field tests showing defects that will rarely appear in pictures. We have uncovered two lenses over the last 5–7 years that really did have issues serious enough that internal factory readjustment was necessary. So finding those early was a big benefit. None of this provides any data about 'my lens is sharper than your lens.' But it does provide confidence that the system can produce excellent results if the photographer doesn't screw them up."

Thom Hogan: "Re 'Exhaustively understands.'

"Well you got that right, it exhausts me. Worse still, it's confusing. For the first time in a long time I'm about to take a long trip with cameras that I can say I've been using for awhile now (okay, one's a new version of an older model, but there's not a terrible lot to learn about it if I ignore the video part ;~). Put a different way: I'm about to take a trip where I can solely concentrate on photography.

"Oh wait, I'm bringing two new lenses  ;~).

"We're at a point in photography where we're picking nits. There isn't a camera on the market that won't take perfectly fine photos if you let it...uh, I mean if you control it.

"That said, what's been said by a few others I'll repeat: one of the reasons why I got into testing my gear extensively before using it in production was knowing what I'm dealing with. I believe that photography is really the collection of optimal data, which shows my Ansel Adams learning. I've found plenty of miscentered lenses, buggy firmware, features to avoid, sensor issues that impact my post processing, etc. I prefer to know about those before a moment comes along that won't happen again.

"I'll give a recent example of frustration. I was shooting 4K video in Botswana this summer with a brand new camera (new to the market, new to me) when that leopard started fishing for catfish in the mud puddles as the Savuti dried up. I can't tell you how many times I hit the wrong button (out of several dozen on that camera) while trying to get decent footage of that unique leopard.

"Fortunately, the leopard was cooperative and just kept repeating her actions over and over. But had there only been one big catfish in the puddle, there's a chance that my button dysfunction might have screwed up the shot.

"One correction to one of the comments: mirrorless is not necessarily self-calibrating for focus. Olympus has AF Fine Tune built into the E-M1 for a reason: if you rely solely on phase detect AF (or Panasonic's DFD) to achieve focus, you need the ability to tune camera and lens together. If the camera also uses contrast detect to tweak the phase detect focus point each time, you're going to slow down the focusing system, especially on continuous AF."

Comments

[Ken Ford]

"...with my friend's X-Pro2"

You may want to correct that unless you're trying to push page views. ;-)

[Who said that? --Mike]

[Herman]

Testing the testers can be as difficult as testing the equipment.

[Bob Rosinsky]

I know a techie gal photographer who tests her tests. She is perhaps the only photographer I know with whom I avoid discussions on photography.

[Roger Botting]

Just don't get carried away too much in testing equipment. You might start getting compared to the 1970's Pop Photo. Testing camera straps, lens cap holders, vinyl film can holders, etc.

[Dave Levingston]

I'm pretty much with you as far as testing new cameras. I sit down and figure out how to use the thing and how to set it up to do what I want it to do. Then I go shoot. I quickly learn how it performs for what I want it to do. Since digital came along I find that each new generation of camera is better at what I want than the previous one. I'm not sure what more I need to know.

[Tony McLean]

Who knows how carefully mice make plans, anyway?

Steinbeck?

[MarkR]

I have often wondered why some people have to return a half dozen copies of a lens before they find a good one, while I have always gotten a perfectly satisfactory one on my first attempt. Just lucky, I guess ...

[Earl Dunbar]

Rabbie Burns always makes me smile. I am hoping my kilt is finished in time for Jan 25, which my friends and I will celebrate on Jan 23. Just enough time to recover for the actual day.

[JTW]

"The best laid schemes of Mikes and men / Often go awry."

Really? That's so bad, it's great.

[David Anderson]

Mike I happen to have the X-Pro 1 and the X-T1 and the main difference I notice in the out of camera JPG (not RAW) is that at high ISO the detail just gets smeared in the X-T1, not so the X-Pro1. Apart from that all is good.

[David Sutton]

With a camera I want to know how far I can push the ISO before the reds and blues look awful, and with a lens what the OOF highlights look like.
There are no "scientific" test for this as my awful may be your frightful. :-)

[Dave Riedel]

Here's an interesting approach to photography and paintings... http://photographersoilcollective.com/

See also the discussion on ownership rights to the finished product.

[Joe Holmes]

"I see too much when I look at pictures already." There's a whole essay in that sentence.

Also: I've always loved the sound of "Gang aft a-gley" (you have to hear it with the accent). The modern English translation loses the poetry.

[Roger Overall]

I don't test lenses anymore. The last time I did, the lens in question ended up split open, filled with sand and buried in a shallow grave in the garden.

You see, I'd set up my camera on a tripod and pointed it at what I thought was a suitable combination of brickwork and vegetation to test the mettle of the gorgeously expensive Canon L glass I had recently acquired.

So curious was I about the results that, instead of dismantling the set up, I popped out the CF card and hurried inside to have a preliminary look on my computer. Upon my return, the tripod was flat on the ground, camera still attached to it. The lens had cracked open in the fall and then been buried by the culprit in a bid to hide their crime. Nearby, my then-three-year-old daughter gave an Oscar-worthy performance of pure innocence as she pottered about.

The repair bill was substantial. So, I've decided that lens testing is just too damned expensive to be worth the effort.

[Kenneth Tanaka]

What Gordon Lewis wrote also reflects my perspective and experiences. If a lens or camera is a stinker or truly defective it will be soon discovered in the course of use.

As I've neither had a photo ruined by a bad lens nor have I seen a photo by others so ruined I don't have the anxiety over optical perfection that so many photographers have. The state of the art is so good, even in very modest budget optics, that I think "testing" is a waste of time. Shoot, man, shoot!

[Bryan Geyer]

It's my experience that most lens testing is fatally flawed by initial failure to assure that the lens-to-target alignment is ABSOLUTELY parallel to the film/sensor plane.

Absolute parallel alignment of lens-to-target alignment is best assured by precise laser spot reflection, using a professional FRONT-SURFACED mirror at the target, a precision (micro-geared) mechanical stage to adjust (3 axis) target positioning, and a laser source that's pre-aligned for accurate (perpendicular) beam output.

It's quite amazing to realize how poor ordinary visual (or simple mirror reflection) alignment can be when you subsequently apply this more precise laser reflection methodology.

[Patrick Dodds]

@ Dave Riedel - not my cup of tea, there used to be a filter in photoshop that would do what they've done to those pictures.

[Mahn England]

You see that's the problem with photography...there are far too many variables.

[Christopher Mark Perez]

I've looked at camera systems for years (I was an engineer/scientist by trade), mainly out of curiosity to see if I could understand what's really going on with imaging.

I'm not sure I have "the answer" for anyone but myself, but it's this - Film (from back when dinosaurs roamed the earth), sensors, and processing are more often than not the limiting factors to image resolution.

It's perhaps hard to believe, but it's true in every single case but one (and I've looked at many hundreds of systems - http://photosketchpad.blogspot.fr/ - look along the right side of the page). Only once have I encountered an obviously bad optic (Canon 17-55 f/2.8 which normally has a decent reputation) and I returned it immediately.

Perhaps the biggest lesson for me to come from all this poking and prodding and looking at things is that commercially available imaging systems are very seldom an impediment to making a decent image. It's the brain and heart behind the eyepiece/LCD that counts.

[Steve Jacob]

There's testing under controlled conditions - in which you override all the defaults to equalise everything - and real world conditions where the effort of doing so is seldom considered.

The first may discover whether there is an actual mechanical issue, the second will decide whether you actually like the darned thing or not.

I'm with Gordon. The biggest bugbear I had with DSLRs was inconsistent focus. It could be very slight, but just enough to be annoying. The more consumer oriented the lens and camera were, the worse the problem was, but I have also had issues with particular lens/camera combinations on top range gear.

Change camera or lens and the issue goes away. Neither is individually broken, but a combination of manufacturing tolerances unique to that pairing just throws everything out. My D800 and 16-35 F4 was a classic case in point.

One advantage of MILC focusing is that it is self-calibrating. Using the sensor to focus is accurate every time, assuming you get a good lock. I think this was the main reason I switched. If I have a consistent focus issue I can assume that there is a fault.

[Mark Hubbard]

Nice selfie..

[Patrick]

Funny, because I sometimes get into testing mode myself, then later will forget whatever it is I learned. My latest was to try to learn what combination of factors gave me the sharpest photo. As in - with mirror lockup or without, using a shutter release or self timer, with or without live view, and all sorts of permutations of the above. (I think live view was the winner, but I probably need to retest to be sure)

[Seth]

The generally high quality of the hardware these days further complicates matters. Even the base level Canon or Nikon + kit lens (or even the right cell phone camera) is capable of very good results under the right circumstances. It's easy to come up with a test that'll distinguish between a coke bottle and a Zeiss; it's a much trickier proposition to design a test that will tease out the differences between an outstanding lens and a merely great one.

[William]

Ad-hoc testing is usually disappointing at best and completely misleading at worst.

In this case the XTRans I data stream's raw files are inherently different than the Xtrans II raw. There is a shift in the EV scale. That is, the EV ranges have different symmetries about zero EV (about a 2 stop shift)

A statistical analysis of unrendered raw data will reveal the XTrans II raw is has slightly improved signal-to-noise and hence dynamic range. This difference is almost too small to be relevant. However the extra bits in the Xtrans II's ADC are not there for purely marketing reasons. The Xtrans II analog SNR is just high enough that a 14 bit ADC can be purposeful.

Another example is the accurate, but incomplete observation that XTrans II detail is smeared (especially at high ISOs) compared to Xtrans I detail. This is so for in-camera JPEGs, but is not the case for raw rendering. The difference is in the X-T1's EXR Processor II vs the X-Pro1's EXR Processor Pro in-camera JPEG rendering. The former applies more aggressive noise filtering at all Sharpness Menu settings.

[Michael Bearman]

It's all relative. I have audiophile friends who have spent an absolute fortune so that they could accurately reproduce the induced distortion of 70s prog rock.

[Stephen Scharf]

I'm with William on his observations.

In my own testing, while I find there to be a small "statistical" difference between the two Xtrans sensors, I don't really find there to be a practical one, for the most.

I do seem to observe, however anecdotally, that black and white conversions from the X-Pro1 have a bit more of that subtle Fuji magic than conversions from the X-T1.

[Brian Stewart]

As per the comments by several other posters, it's my understanding that there are two versions of the JPG engine. The differences you observed in the second test are well explained by the different JPG engines. Did you compare RAFs? The benefits of the higher bit depth of the X-T1 are unlikely to be seen in this shot, and so I'd expect them to be virtually identical, in whichever RAW converter you happened to choose.

Personally, I'd like to have JPG engine from my X-E1 in my X-T1.

[Robert]

Kodak got it, Olympus for a brief moment we're perfect and Minox came close. It's not about the lens, it's about what the customer wants and needs. Responding to this is one part of the marketing process. The lens obsession was never a criterion for me. As a business traveller, I needed a camera I could take anywhere and always use it. This was many years ago and I tried almost everything from a Rolleiflex TLR down. My need was to be able to have a camera that would fit in my briefcase when I was on a day trip to Europe or, on a longer trip but with a small bag. There were so many photographic opportunities that I could use that I wanted a camera that I could access quickly and surreptitiously and snap away. The small OM's were very good but then they produced the Oyster Shell, pocket sized Olympus - perfect, I used it for years, even snapping some great shots in the cockpit of 747's, which was welcome in those days. It went everywhere and was not an intrusion. I also had the small Minox with the folding front. Met the criteria but was not quite so snappy to use.
So, just to be contoversial I would ask if obsession with lens performance is not a bit too nerdy for most photographers - as Jim Richardson said the camera has to be suited to your needs. Given that for most of us the lens quality across almost all cameras is indiscernible, especially when we can post process it out of existence, why is it so important? Does anybody complain about lens performance of an iPhone when it is even stuck in a pocket, has fingers all around it. You can probably take a really cheap phone camera, swish its photos through almost any processor and produce as good a photo as a real camera with a real lens.
P.S. I am not a fan of phone cameras.

[David Dyer-Bennet]

Since consistency and quality control on the manufacturing line is one of the main challenges in making lenses, testing one sample never tells us anything all that conclusive about a lens design. Essentially everywhere that does lens testing has this problem. I suppose it does, at least, give you a hint of what the lens can be capable of at its best. (Consumer Reports seems to be the one consumer testing group that actually understands sample variation and statistics; but what they test for in computers and cameras are more relevant to ordinary consumer use than to my own professional or serious hobbyist use.)

[Tom Bell]

Weirdly i like what Fuji do with Bayer sensors

Love my X100 and use a XA1 with my other Fuji lenses.

Hope they will bring out an XT1 with a Bayer Sensor!

[Miserere]

My attitude towards (not) testing lenses, as I wrote about back in 2010:

I Don’t Care Much about Comparing Lenses/

Surprisingly, it wasn't the death of my blog :-)