Roger Cicala said you're never supposed to start a commentary with the words "Roger said." But in this case it's a direct quote, so I'll chance it...Roger said, "I'm not a lens reviewer; I just test the MTF. There's so much more that goes into choosing a lens."
That's the first "meta" complication in lens testing: namely, that each test describes what the testing protocol tests. I stated this crudely years ago when I mentioned that guys who liked to test lenses by taking pictures of sheets of newspaper taped to the wall are testing how good their lenses are at shooting newspapers taped to walls...i.e., small-scale flat-field subjects at near distances with detail of a certain frequency. (I added that that's assuming their testing skills are rigorous, which they often aren't, and that the tester has the knowledge to attribute characteristics properly, e.g., which types of error are caused by which aberration in the lens.)
But even beyond that, there are further complications that are given way too little consideration out in the wilds and warrens of the Internet.
The first is perceptual testing. What this means is gathering data about what visual characteristics people subjectively like, and relating those visual characteristics to the optical properties of the lenses that produce them. This was done comprehensively by Loyd [sic] Jones and C.E.K. Mees of Kodak in the 1930s and '40s in determining what kind of contrast people liked to see in prints—it led to the system that pertained for many years of targeting development of film to a standard contrast index (C.I.) and then further refining contrast by mean of paper grades—and what kind of characteristics people most preferred in lens images.
I've done this kind of testing myself several times (in the years around 1990, mostly), and it can be fascinating. I actually loved doing it and would have loved to have done more, but it's very labor-intensive, time-consuming, and costly for one unfunded individual to do*. Just as an example, one test I did compared prints made from a 4x5 negative with prints of the same subject made from a 6x7 negative. I showed pairs of prints at various sizes to a sampling of people and asked if a) they could tell a difference and, if so, b) which they preferred. What I learned in that specific case were two things I felt comfortable generalizing: 1.) that there was a point where the prints were small enough that the difference didn't matter and there was a point where the prints were large enough that it did matter, allowing you to extrapolate very roughly as what size enlargement the distinction begins to become meaningful; and 2.) photographers were able to detect the differences more easily than visually sensitive non-photographers could. The reason is that they knew which clues to look for. The point here is just that perceptual experiments are both fascinating and illuminating; they attempt to measure not just whether a lens is good but in why it's good—or you might put it, in what ways good is good. It's not a form of experiment anyone ever attempts today, that I know of.
Mees and Jones were designing the famous Commercial Ektar series of professional view camera lenses, and their experiments resulted in those lenses being optimized for high large-structure contrast rather than resolution, a design choice that carried over into certain early Leica lenses (and a choice I still prefer, personally, in B&W).
Generally speaking, what this indicates is that the choices one makes as to what constitutes "goodness" in the first place is also provisional and ultimately arbitrary. The best proof-of-concept of that claim is to look at lenses from the Pictorialist period (1885–1915, roughly). The properties that were prized and desired then, soft focus and low contrast, were the opposite of what is prized and desired today. I'm not saying either is better or worse, merely pointing out that the technical parameters we like to measure and value are not eternal values written in stone, but are actually just choices we make based on subjective agreement...often merely assumed and not particularly taste-driven. At least not in accordance with the taste of any one individual practitioner. I have a friend who is a great optical connoisseur who deplores overly-perfected lenses; he uses words like "clinical" and "sterile" and "synthetic" to describe just the sorts of lenses that would ace MTF tests. (He's hard to please; I believe one lens he approves of is the Leica 35mm Summilux ASPH. Type I, which he owns.)
This leads to a complication that is even further removed, namely, how suitable the lens rendering is to what the operator (artist/photographer) is trying to express. I think it's fair to say that there is a minor "tradition" among working art photographers of taking in the nit-picking that exists around the analyses of lenses and then "going the other way" so to speak, by deliberately choosing lenses that are "poor," according to the common consensus, and seeing what can be done with them. There are many examples, from photographers who used Diana or Holga plastic toy cameras or cracked or damaged lenses, to those who made virtues of any (or many) of a whole variety of what are normally considered "flaws," whether that might be distortion or vignetting or flare or what have you. The look of flare can be so interesting aesthetically that it's been adopted in other media, for instance in Georgia O'Keefe's The Shelton with Sunspots, N.Y. (1926) or the Pixar animated feature Ratatouille (2007).
Tech culture changes too
Of course, the very meaning of the idea of "lens performance" is changing. Case in point, in case a case in point is needed, are those lenses in which distortion is uncorrected in the glass because it is intended to be corrected later in software. Vignetting is also obviously no longer an issue, because it is so easily corrected (or exaggerated) in post. So what I've been saying for thirty years is become more evident everywhere in photography: technical properties are more important as aesthetic qualities. While there are some obvious common goals among technical properties, there's no stable definition of "good" when it comes to aesthetic qualities.
Most importantly: what is defined and accepted as "good" among technical properties does not equate to "good" among aesthetic qualities. Rather, the look in pictures of a functionally perfect lens, while it might be a sort of holy grail to techie geeks online, is aesthetically just one more quality among many. Expressively it suits some pictures (and photographers) while not suiting others.
My observation over many years has been that photographers of sensibility typically work toward the look they want until they get there. This can mean anything from "settling" quickly for whatever common equipment is on hand, to working for years to refine an esoteric process that might even eventually become identified with them (Sheila Metzner and Phil Borges are two examples of the latter who come to mind). Now this is coming to include Photoshop post-production skills and choices as well. And of course we are always free to intentionally "misuse" equipment to exploit the aesthetic qualities of what are technically considered faults. An example that comes to mind might be Frederick Sommers' out-of-focus pictures.
All this is taste-based, not tech-based. So what I'd recommend is not necessarily choosing your lenses based on lens tests at all. Information is good, and good information like Roger's is infinitely better, but even good information is still just data. A good lens is one that does what you want it to do and that you feel good about...based on how the pictures look. Keep looking until you get there—whether finding it comes early or late, and whether the process is easy or difficult, casual or fanatical, or expensive or cheap. Don't let other people tell you what you're supposed to like, yes; but be mindful not to let lens tests dictate to you what you're supposed to like either.
*A later experiment involved sending ten prints of ten different pictures to volunteers to see whether they could identify the two that were taken with Leica lenses. But the volunteers were so contentious and captious that I quickly gave that one up!
Original contents copyright 2017 by Michael C. Johnston and/or the bylined author. All Rights Reserved. Links in this post may be to our affiliates; sales through affiliate links may benefit this site.
(To see all the comments, click on the "Comments" link below.)
Featured Comments from:
Roger Cicala: "Roger Cicala says it is not only acceptable for Mike Johnston to start comments with 'Roger said,' it is absolutely flattering. :-) "
Pieter: "And even after doing many (possibly flawed) tests, I can (somewhat) objectively and subjectively show that my 28–300mm Tamron lens is worse in almost every way than a set of primes, or a series of narrower zooms. Yet somehow it's the lens I use the most, purely out of convenience when traveling light, and some of my best photos were taken with it!"
John Krumm: "I enjoyed the article. Lens chasing can be a bit like Pinot Noir chasing. You've had that one good bottle, and you know there must be another somewhere... My Pentax 31mm Limited at times seems sublime, at other times flawed. I suspect the flawed part has to do with field curvature, but really I have no idea. All my 4/3 and Micro 4/3 lenses, zooms and primes, are slight variations of really good to excellent."
Herb Cunningham: "Amen: why fans of 50mm lenses have more than one, and in my LF days, why the old Dagors and Petzval lenses were popular."
Kenneth Tanaka: "I very much enjoy Roger's analyses and tear-downs, especially his zoom lens analysis. I admit to not absorbing all of it but I did get its gist. That said, I think the fundamentals of lenses and their role in photography have dramatically changed in the digital era. In the chemical era the lens had perhaps the greatest influence on the final image for most work. So the degree to which a lens offered the nuanced inflections you outline was very important, coupled, of course, with its other geometric and sharpness properties. In today's digital era the lens is almost purely utilitarian, the pipe that delivers light to the sensor. Nearly all lenses of contemporary design and manufacturing are good enough for most digital work. The character of images is added later, along with gentle corrections for design compromises. That is, the lens is merely the first word in today's digital image, and rarely as influential as its chemical-era predecessors."
Mike replies: Very well said (briefly, too). I think that's a fair assessment and for the most part I agree.
...At least, it's not nearly as much fun being a lens geek as it used to be!