A Few Further Complications In Lens Testing

Roger Cicala said you're never supposed to start a commentary with the words "Roger said." But in this case it's a direct quote, so I'll chance it...Roger said, "I'm not a lens reviewer; I just test the MTF. There's so much more that goes into choosing a lens."

That's the first "meta" complication in lens testing: namely, that each test describes what the testing protocol tests. I stated this crudely years ago when I mentioned that guys who liked to test lenses by taking pictures of sheets of newspaper taped to the wall are testing how good their lenses are at shooting newspapers taped to walls...i.e., small-scale flat-field subjects at near distances with detail of a certain frequency. (I added that that's assuming their testing skills are rigorous, which they often aren't, and that the tester has the knowledge to attribute characteristics properly, e.g., which types of error are caused by which aberration in the lens.)

But even beyond that, there are further complications that are given way too little consideration out in the wilds and warrens of the Internet.

Perception
The first is perceptual testing. What this means is gathering data about what visual characteristics people subjectively like, and relating those visual characteristics to the optical properties of the lenses that produce them. This was done comprehensively by Loyd [sic] Jones and C.E.K. Mees of Kodak in the 1930s and '40s in determining what kind of contrast people liked to see in prints—it led to the system that pertained for many years of targeting development of film to a standard contrast index (C.I.) and then further refining contrast by mean of paper grades—and what kind of characteristics people most preferred in lens images.

I've done this kind of testing myself several times (in the years around 1990, mostly), and it can be fascinating. I actually loved doing it and would have loved to have done more, but it's very labor-intensive, time-consuming, and costly for one unfunded individual to do*. Just as an example, one test I did compared prints made from a 4x5 negative with prints of the same subject made from a 6x7 negative. I showed pairs of prints at various sizes to a sampling of people and asked if a) they could tell a difference and, if so, b) which they preferred. What I learned in that specific case were two things I felt comfortable generalizing: 1.) that there was a point where the prints were small enough that the difference didn't matter and there was a point where the prints were large enough that it did matter, allowing you to extrapolate very roughly as what size enlargement the distinction begins to become meaningful; and 2.) photographers were able to detect the differences more easily than visually sensitive non-photographers could. The reason is that they knew which clues to look for. The point here is just that perceptual experiments are both fascinating and illuminating; they attempt to measure not just whether a lens is good but in why it's good—or you might put it, in what ways good is good. It's not a form of experiment anyone ever attempts today, that I know of.

Mees and Jones were designing the famous Commercial Ektar series of professional view camera lenses, and their experiments resulted in those lenses being optimized for high large-structure contrast rather than resolution, a design choice that carried over into certain early Leica lenses (and a choice I still prefer, personally, in B&W).

Generally speaking, what this indicates is that the choices one makes as to what constitutes "goodness" in the first place is also provisional and ultimately arbitrary.  The best proof-of-concept of that claim is to look at lenses from the Pictorialist period (1885–1915, roughly). The properties that were prized and desired then, soft focus and low contrast, were the opposite of what is prized and desired today. I'm not saying either is better or worse, merely pointing out that the technical parameters we like to measure and value are not eternal values written in stone, but are actually just choices we make based on subjective agreement...often merely assumed and not particularly taste-driven. At least not in accordance with the taste of any one individual practitioner. I have a friend who is a great optical connoisseur who deplores overly-perfected lenses; he uses words like "clinical" and "sterile" and "synthetic" to describe just the sorts of lenses that would ace MTF tests. (He's hard to please; I believe one lens he approves of is the Leica 35mm Summilux ASPH. Type I, which he owns.)

Expressiveness
This leads to a complication that is even further removed, namely, how suitable the lens rendering is to what the operator (artist/photographer) is trying to express. I think it's fair to say that there is a minor "tradition" among working art photographers of taking in the nit-picking that exists around the analyses of lenses and then "going the other way" so to speak, by deliberately choosing lenses that are "poor," according to the common consensus, and seeing what can be done with them. There are many examples, from photographers who used Diana or Holga plastic toy cameras or cracked or damaged lenses, to those who made virtues of any (or many) of a whole variety of what are normally considered "flaws," whether that might be distortion or vignetting or flare or what have you. The look of flare can be so interesting aesthetically that it's been adopted in other media, for instance in Georgia O'Keefe's The Shelton with Sunspots, N.Y. (1926) or the Pixar animated feature Ratatouille (2007).

Tech culture changes too
Of course, the very meaning of the idea of "lens performance" is changing. Case in point, in case a case in point is needed, are those lenses in which distortion is uncorrected in the glass because it is intended to be corrected later in software. Vignetting is also obviously no longer an issue, because it is so easily corrected (or exaggerated) in post. So what I've been saying for thirty years is become more evident everywhere in photography: technical properties are more important as aesthetic qualities. While there are some obvious common goals among technical properties, there's no stable definition of "good" when it comes to aesthetic qualities.

Recommendation
Most importantly: what is defined and accepted as "good" among technical properties does not equate to "good" among aesthetic qualities. Rather, the look in pictures of a functionally perfect lens, while it might be a sort of holy grail to techie geeks online, is aesthetically just one more quality among many. Expressively it suits some pictures (and photographers) while not suiting others.

My observation over many years has been that photographers of sensibility typically work toward the look they want until they get there. This can mean anything from "settling" quickly for whatever common equipment is on hand, to working for years to refine an esoteric process that might even eventually become identified with them (Sheila Metzner and Phil Borges are two examples of the latter who come to mind). Now this is coming to include Photoshop post-production skills and choices as well. And of course we are always free to intentionally "misuse" equipment to exploit the aesthetic qualities of what are technically considered faults. An example that comes to mind might be Frederick Sommers' out-of-focus pictures.

All this is taste-based, not tech-based. So what I'd recommend is not necessarily choosing your lenses based on lens tests at all. Information is good, and good information like Roger's is infinitely better, but even good information is still just data. A good lens is one that does what you want it to do and that you feel good about...based on how the pictures look. Keep looking until you get there—whether finding it comes early or late, and whether the process is easy or difficult, casual or fanatical, or expensive or cheap. Don't let other people tell you what you're supposed to like, yes; but be mindful not to let lens tests dictate to you what you're supposed to like either.

Mike

*A later experiment involved sending ten prints of ten different pictures to volunteers to see whether they could identify the two that were taken with Leica lenses. But the volunteers were so contentious and captious that I quickly gave that one up!

Original contents copyright 2017 by Michael C. Johnston and/or the bylined author. All Rights Reserved. Links in this post may be to our affiliates; sales through affiliate links may benefit this site.

Not cheap but cheapER
Nikon too

(To see all the comments, click on the "Comments" link below.)
Featured Comments from:

Roger Cicala: "Roger Cicala says it is not only acceptable for Mike Johnston to start comments with 'Roger said,' it is absolutely flattering.  :-) "

Pieter: "And even after doing many (possibly flawed) tests, I can (somewhat) objectively and subjectively show that my 28–300mm Tamron lens is worse in almost every way than a set of primes, or a series of narrower zooms. Yet somehow it's the lens I use the most, purely out of convenience when traveling light, and some of my best photos were taken with it!"

John Krumm: "I enjoyed the article. Lens chasing can be a bit like Pinot Noir chasing. You've had that one good bottle, and you know there must be another somewhere... My Pentax 31mm Limited at times seems sublime, at other times flawed. I suspect the flawed part has to do with field curvature, but really I have no idea. All my 4/3 and Micro 4/3 lenses, zooms and primes, are slight variations of really good to excellent."

Herb Cunningham: "Amen: why fans of 50mm lenses have more than one, and in my LF days, why the old Dagors and Petzval lenses were popular."

Kenneth Tanaka: "I very much enjoy Roger's analyses and tear-downs, especially his zoom lens analysis. I admit to not absorbing all of it but I did get its gist. That said, I think the fundamentals of lenses and their role in photography have dramatically changed in the digital era. In the chemical era the lens had perhaps the greatest influence on the final image for most work. So the degree to which a lens offered the nuanced inflections you outline was very important, coupled, of course, with its other geometric and sharpness properties. In today's digital era the lens is almost purely utilitarian, the pipe that delivers light to the sensor. Nearly all lenses of contemporary design and manufacturing are good enough for most digital work. The character of images is added later, along with gentle corrections for design compromises. That is, the lens is merely the first word in today's digital image, and rarely as influential as its chemical-era predecessors."

Mike replies: Very well said (briefly, too). I think that's a fair assessment and for the most part I agree.

...At least, it's not nearly as much fun being a lens geek as it used to be!

Comments

[Anthony Shaughnessy]

Looking at the Alfred Stieglitz collection on-line, linked to from Ken's post, has re-inspired me with regard to the look of soft lenses and made me think about how I could break out of my normal way of shooting by trying some of my older lenses that I don't normally shoot with anymore.
Anthony

[Mike Aubrey]

Good discussion.

It reminds me of a pdf I encountered a while back:

"The Pinhole Camera, Or the Revenge of the Simple Minded Engineer"

https://www.kth.se/social/files/542d2d2df276546ca71dffaa/Pinhole.pdf

[David Babsky]

In the last days of Olympus film cameras, I could always (..seems unbelievable, I know..) tell - of the pictures which I saw, anyway - which pics had been taken with Olympus lenses - or Oly cameras with non-interchangeable lenses - because they were so contrasty, and thus appeared so very tack sharp, compared with other brands ..whereas Nikon and Minolta pics looked "soft" by comparison ..though Nikon and Minolta were far "better", or more appropriate, for shots of misty lakes, as they delivered the mistiness with exquisite detail ..which the more contrasty Oly lenses didn't do: instead of a full range of tones, the Oly lenses delivered harsher, contrastier pictures.

Oly for punch, Nikon and Minolta for contemplation. Canon, Pentax; in between. It all depended on what you wanted..

[john robison]

Don't forget cost. As in, Gee, I keep finding I cannot back up enough, I need a wider angle lens. (Goes online to check out wide angle lenses) Cripes! They sure are expensive. Lets see, what can I get for a $200 or so....well, I could buy this third party prime, used, for $150. Ok, that'll be good enough.

Decision made based on what you can afford. You have no other option but to be happy with the results. See?

[Richard Jones]

In my 40+ years of photographing, I have rarely looked at a lens test, and have never bothered with test charts.

If I'm interested in a lens, I'll purchase (sometimes rent) the lens and photograph what I like to photograph and evaluate accordingly.

One of my favorite quotes:

"Careful photographers run their own tests." — Fred Picker

Richard

[Rick in CO]

I agree, and I have taken some very nice pictures with top tier Canon and Nikon zoom lenses. But isn't Roger's take home message that zooms are optimized for a particular chosen focal length? At other than the optimal focal length one sacrifices some amount of image "quality" for convenience?

[Pierre Charbonneau]

You have nailed it again Mike.

I plead guilty to the perpetual quest for the "lens" that will convey just the right amount of character, of seductive and elusive bokeh and the appropriate snap when required. Many Summicrons, Summiluxes, and Elmars have been dearly purchased, cherished for a while and later exchanged often without much regret.

Funny thing now I have settled for a 1969 Summicron 35 V2 and a 1984 Summicron 50 V4, I like them because they are small and light. And allright optically.

[Michael Perini]

Mike, everything you say is true, and like most other things the variables involved in the technical and aesthetic characteristics of lenses can approach the infinite.
But on a practical basis, most folks filter some of that out in the selection process. Some are drawn to contrast, some to resolution , some to particular aspects of a lens's character.What is right for you, is right for you, period.
However it can no longer be disputed that copy to copy variability exists even among the finest lenses.
So I believe that prudence dictates that at a minimum you design some repeatable tests that will confirm that the lens is not an outlier. That it will perform in a way that comports with the reasons you chose it.
Some may want to go further, but with lenses costing as much as they do, setting up a repeatable protocol for 'testing' that is informative without being onerous seems to make sense.
The fact that there can be dozens of other factors, many of which might require specialized knowledge or equipment, shouldn't deter us.

[Eric Erickson]

Mike, I really liked what you had to say today and what Roger said. My favorite lens photographically is the Nikon 24-70 f2.8. It just provides the best images I have ever taken; however I rarely use it lately since I got the small Fuji camera, because the lens and FF Nikon are too big and heavy. I find that despite the fact that the Fuji does not produce the quality of images that my Nikon does it still goes with me 90 percent of the time. I simply make it work. You make the lens and camera you have do the work you need done. Photoshop and Lightroom do help a lot but most lenses today are really "good enough".

[Patrick Perez]

This reinforces a thought I've had, which is that the vast majority of photographers would be best served not chasing a lens of specific optimization (contrast, sharpness etc) since as you and Roger demonstrate in different ways, lenses perform differently depending on subject distance, enlargement etc etc and rarely are the optimizations optimal in all conditions. Photographers who shoot very specific subjects under standard conditions might benefit from seeking a holy grail performing lens, but most folks, I say, should get a good all 'rounder and worry less.

Patrick

[tex andrews]

"My observation over many years has been that photographers of sensibility typically work toward the look they want until they get there."

Bang. Nuff said.

[Gordon Lewis]

I find that it's not a lens's perfections that make it interesting but rather its idiosyncracies. (I won't call them flaws.) Sometimes it's the lens you use that informs your style. Sometimes it's the reverse. Either way, it's when the two are intertwined that you know you've got something good cooking in the pot.

[John]

Hi Mike
You may be interested in this fascinating and detailed bio of Mees from the New Scientist in 79. Thanks for all the many thought provoking posts.
John

[John]

https://books.google.co.uk/books?id=3P5-7ZAAXsYC&pg=RA1-PA953&hl=en#v=onepage&q&f=true
Sorry here is the link to the Mees bio.
John

[Bear.]

Quite interested in your print perception test. I would like to see similar tests with different sensor sizes on large prints.

[BH]

Interesting how all those "flawed" older lenses have more of a look, at least in certain conditions. Sometimes I want my photographs to look like photographs, you know? Other times I couldn't care less.

What's this race for optical perfection gotten us? Behemoth 50mm's that are awesome at f/1.4 and cost an arm and a leg. Great. What are you putting in front of it?

[Steve Jacob]

I think testing will tell you a lot about what's wrong with a lens, but less about what's right with it. It's obvious from a newspaper test when a lens has a major decentering issue, but you can't tell much about its general performance if all the 'flaws' are the same in each corner.

But I admit to being a heathen. I want a lens that gets out of the way and doesn't deny me access to the image. That's equates to a lack of character. I want clinical sharpness, good flat-field performance and good OOF transition.

The rest I can take care of. Even a bit of regular distortion and CA doesn't bother me because its much easier to remove than coma. One of the reasons I am so enamoured of lenses like Fuji's XF 14 2.8 is that it is almost vice-free, which is unusual in a wide-angle lens.

Aesthetics is something I can control if the hardware just stays out of my way.

In that sense I think 'character' is almost meaningless in a digital context. We are now dealing with a GIGO problem. Garbage in, garbage out.

[Paulo Bizarro]

Excellent write up. I think that today, with advanced computer aided lens design, where results from lenses can be simulated a priori, the technical aspect is more than good for everybody. Then comes the personal perception, which is more subjective, and is more important to me.

I think that today it will be really difficult to buy a technically poor lens, from the well known and established makers.

[Pieter]

And even after doing many (possibly flawed) tests, I can (somewhat) objectively AND subjectively show that my 28-300 Tamron lens is worse in almost every way than a set of primes, or a series of narrower zooms. Yet somehow it's the lens I use the most, purely out of convenience when travelling light, and some of my best photos were taken with it!

[Matt]

I don't get this obsession with lens testing. Perhaps the most boring aspect of photography is pseudo scientific lens testing. It doesn't tell you anything that helps you make good pictures.

I do notice the difference in lenses, but I don't need a dorky test to tell me this. There's a lot subjectivity and impression in play that cannot be captured in MTF scores.

Just take pictures and decide what works for you.

[Michael Perini]

There are parallels to the music industry.
First we had bad digital then we got to good digital,
But every studio in the world keeps a Mic locker with character full mics,
And racks of analog outboard gear to warm up the sound (add even order distortion)
All tastes can be accommodated from super clean to grunge.
For the same reasons vinyl retains a small but loyal following.
Perfection seems to make us uncomfortable.

[Peter Wright]

There is obviously a ton of information just below the surface here. If you were to amplify and include examples, this could become book length or at least an extended series of articles. Perhaps you have that in mind?

[Richard Howe]

To begin with I was enough of a gear head that I avidly read lens tests and never enjoyed any of them more than an I enjoyed that comparison of test results on what was it? eight? different copies of the same lens model. (That was Roger, right? — too lazy to look it up.)

But that's not how I came to have the lenses I now have. Instead, I got them, in the first place, when I was seized with the photography mania ("addiction"?) because somebody whose judgement I knew and respected to me that they were "the best."

Been the curious sort, I also bought a lot of other lenses in those first few years (when I was still pretending to be rich), and was of course fascinated by the differences I could see in the results.

So, to try to get a handle on those differences, for a couple of years I photographed, over and over again, the building on the southeast corner of East 12th Street and Second Avenue, less than a block from where I was living in Manhattan at the time.

I always stood in the same place on the diagonally opposite corner and always centered the frame on the same place (more or less) on the building. Sometimes I used a tripod, sometimes not.

I wasn't at all systematic about it: I did this whenever I had a new lens or camera to try.

I took I don't know how many hundreds of pictures of that building, in all seasons of the year, times of day, and weathers (except pouring rain).

Eventually I really got to know the north and west facades of that building very well! And, as you said, Mike, I developed some skill at photographing it.

The lenses ranged from 24mm to 200mm; the cameras from my first, an Olympus C2000, to series of Canons, and even a Leica R8 and an R9, shooting Kodachrome, Ektachrome, Provia, and Velvia.

I also tried various raw developers, but mostly used PhotoShop ACR (I still use it, but I also use PhotoNinja and DxO, depending on what I'm after, what problems the raw image may have, or whatever).

And tried out different choices in color balancing, noise reduction, sharpening, saturation, etc.

I printed many but by no means all of the images, usually 12" wide by 18" tall, and tried different papers also.

Here's what I found out:

I liked some of the lens/camera (or lens/camera/film) combination more than others; ditto for raw processors; ditto for processing strategies; ditto for papers.

And eventually I sold all but three of the lenses (which I still have, and one of which is one of the ones first recommended to me when I was starting out), and I'm down to just one camera).

The lenses are a 50mm, a 28–90mm zoom, and an 80–200mm zoom. I kept them because, well, I liked what I got with them more often than what I got with the other lenses, and not only photographing that building on the corner of East 12th and Second Avenue.

All these "tests" (I just thought of them as "tryings out") were rigorous in just one respect: ultimately I was only interested in finding out what it was, among the possibilities available to me, that I really liked.

I found out, and a dozen years later I still like what I liked then (which isn't to say that I might not like something else better — I still read the occasion lens review … )

[Tom Kwas]

I learned a long time ago with my Hasselblads, that sharp isn't always sharp. I traded a 50mm "C" lens, for a new 50mm "CF" lens, back in the mid 80's when I was redoing my Hasselblads, and was appalled at the quality difference, the "new" 50mm "CF" being noticeably less sharp than the old lens. Hasselblad refused to take the lens back, saying it met "specs", then they felt the need to redesign it twice as the CF FLE, and then the CFi FLE (which didn't even use 60 bay filters, so blaaa...).

What amazed me about the lens, is that it didn't look "Zeissy". You could pick the difference out from my 80 and 150, and 250. Up to that time, I always loved Zeiss, and thought regardless of sharpness, they all had the same "look". Guess not.

I can certainly say to all those on here that say they've never looked at or tested a lens; that as a professional, I tested all my lenses, so that once I felt that I accepted them, I never thought about them again. I cannot quantify what I was looking for tho, I tended to like sharp better than contrasty, and maybe that's what I was looking for.

I will say tho, that I always felt my 50mm CF Zeiss was a victim of the "thinking" that peoples acceptability of sharpness, was based on actual sharpness plus contrast. I remember that the old Modern Photography measured a lens for sharpness in delineating "line pairs", and contrast; which I liked. When MTF curves came along, it seems like poor sharpness could be made up for, with high-contrast; which I did NOT like.

I remember testing a German lens on a model back in the 90's, and when she saw the transparencies, she told me: "...don't even use that lens on me again...".

[Richard]

In the spirit of the post about the Stieglitz collection a few days ago, I say bring back Pictorialism and then we won't have to worry about lens "quality."
We're long overdue for the artistic poles to switch places, anyway.