All Lens Tests are Wrong

By Mike Johnston

The way to test a camera lens is to take pictures with it.

Take lots. Then look at them. If you want to, notice what the the lens does: what it does right and what it does wrong, what you like about it and what you don’t. If you see a lot you like and not much you don’t, then it’s a good lens. If you can’t find anything wrong and everything seems right, then it’s a great lens…for you. It might still have faults—but if they’re invisible in your work, they don’t count.

Don’t think viewers of your pictures will know or care. Most people don’t look at optical quality when they look at pictures. They look at the pictures. In this sense they may have a big advantage over the typical photographer.

Gazing appreciatively at a print of a flower, the proud photographer may be thinking, "look at that superb edge contrast between the red and the green—and what beautiful gradations in the reds!" But his non-photographer friend standing next to him is thinking, "That's a geranium."

The United States (at least) is a nation of shoppers. We love to fret and worry over fine points of distinction between similar products. The search for a "best" this or that becomes a pastime, and it’s a personal thing; sometimes peoples’ identities get wrapped up in it. Once made, our choices evidently need to be defended, sometimes hotly. The product of choice is exalted; competing products are disparaged.

But that's nonsense. A waste of time and of emotional energy. Empty disputation.

Nancy Rexroth, My Mother, Pennsville, Ohio, 1970. From the book Iowa. Taken with a Diana, an ancestor of the Holga.

A talented photographer with a cheap normal lens can shoot rings around a tyro loaded to the chin with all the latest gear. People can do good or bad work with equipment of any description. That goes both ways: you can do wonderful work with a cheap-crap plastic camera (I’ll trot out my favorite example, Nancy Rexroth’s lyrical book Iowa), and you can do perfectly wretched work with the world’s most expensive Linhof and hand-picked Apo lenses. It can be fun to talk about lenses, pore over tests, look at MTF charts we may not even understand, compare single-number "scores" of various lenses—it just doesn't have much of anything to do with photographing. The collective mental effort that is put into all this endless happy shopping by photography enthusiasts worldwide (or webwide) is largely wasted.

Worst of all are the single-value grades or scores. You know—this lens is a B+, that one only a B. This lens gets a 9.7 out of ten on the "love" scale, but only a 9.1 otherwise. This lens’s MTF is a 4.1 (at least in the last case we know what they’re grading, sort of). This lens can resolve X lp/mm, but only if you use a surveyor’s tripod or a gyroscope and Tech Pan developed in goat urine. This lens gets our highest award. That one is is inexpensive and it’s made by the wrong company, thus it’s not worth looking at.

Clarence H. White, Rest Hour (Columbia Teachers' College). The pictorialists at the turn of the 20th century prized soft-focus, low-contrast lenses. How would such lenses "test" in a photo magazine today?

Understanding any lens is a matter of understanding a great number of properties and qualities. Size, weight, target selling price; angle of view, optical falloff and physical vignetting, and QC; resolution, contrast, consistency center to corner, consistency up and down the aperture range, consistency back and forth in the focus distance range; maximum aperture and diffraction; and aberrations. In all, there may be twenty or thirty parameters of technical description that have to be taken into account—and that’s just the basic list, before one begins to get into arcana. The point is that when you take a complex description and boil it down to a single index, you have to decide subjectively how to give weight to various factors. And most lens testers don’t tell you what factors they judge, how they judge them, or how they weigh them. So they’re really telling you…nothing. Actually, it’s worse than nothing, because they’re passing judgment without giving you any data.

How you describe the way your lens makes your pictures look will depend on how you use it. If you shoot slide film, you may be concerned with color transmission characteristics—but if you shoot black-and-white you won’t be bothered much about that. If you often shoot wide open in the near field, then you may be very concerned with bokeh (out-of-d.o.f. blur). But maybe you shoot stopped down and take pictures of objects at infinity distance, and nothing’s ever out of focus in your work. And what taking distance is the lens optimized for? I once tested a certain OM Zuiko against a certain Zeiss Planar. The Zuiko was better than the Zeiss at near distances, but the Zeiss was better than the Zuiko at infinity. So at what taking distances do the testers test? They sometimes don’t tell you, and they almost never test a range of distances. And at which distances do you mostly shoot?

If you are an artist, you can take a lens with any properties and make art with it. Is flare bad? Not if you can make it expressive. Is low resolution bad? Not necessarily—gestalt experiments make it clear that the human brain can detect images in incredibly simple figurations (the infamous smiley face, for example [see below]), or even things that contain no purposeful images at all (clouds, fire, whorls in woodgrain). Is low contrast bad? It wasn’t to the Pictorialists. And so on. I once had a picture that was both published and exhibited that I took with an ancient Kodak Instamatic 104, on color film, which I then printed on black-and-white paper. Is the Instamatic lens a bad lens? I guess it is. But not for that picture. 

Three birds. Photographer unknown.

In fact, if you "test" a lens for any specific purpose, then what you see is…how it performs in that particular circumstance. If you photograph a newspaper taped to a wall to "test" a bunch of lenses, you will have learned only one thing for sure: namely, which of the lenses you’re testing is best at photographing newspapers taped to walls. And even that presumes that you’re careful enough to avoid error, which many photographers don’t have the expertise to be—even slight focusing errors, for one thing, can completely skew your results. At any rate, at best it’s a case of only one of the blind men describing the elephant.

Different lenses can look effectively the same in certain situations. If you take a Pentax 43mm Limited and a Yashica 50mm ƒ/1.9 in Contax-Yashica mount and point them both away from the sun and stop them down to ƒ/8, the image quality of both will be superb. But simply open them to ƒ/4 and turn around and point them "contré jour" (literally “against the day,” that is, the light), and the differences will be significant (the 43mm Limited is exceptionally flare-free and the Yashica flares badly when sunlight hits it).

Identical lenses can also be very different for different photographers, or in different situations. Consider the case of a German photographer I once knew of who specializes in aerial photography of geographic landforms. He photographs at dawn and dusk because glancing sunlight brights out subtle features of the topography more readily. So when he uses a 50mm lens he uses it wide open, and at a taking distances of more than half a mile. A portrait photographer who uses the same 50mm lens for small group shots in studios is always pumping light onto the scene with strobes, has the lens well stopped down, and seldom gets farther from her subjects than about 15 feet, and more often six to eight feet. So what are these two comparing when they compare opinions about one particular lens?

Finally we have to consider how the lens holds up as a physical device. A certain brand of lenses I won’t name has on occasion perpetrated lenses that measure well when new but don’t hold up well over time. So if you buy the absolute best lens based on paper shopping, how sure are you that in three years the lens will still be performing the way it did when it was new?

How robust is it? We’ll assume that most of us don’t drop lenses very often, but I’ll bet most of us have dropped one once or twice. (I have.) Is an occasional shock going to knock the thing cockeyed so that it never works right again? I recall many years ago hearing about a Leica rep who scraped the front element of a lens with barbed wire, to no effect. Then, as if bowling, he sent the lens skittering across the floor until it banged into the baseboard. He picked it up, clicked it on to the camera, and said simply, "You go on shooting."

The bottom line is that no one else can tell you how good your lens is. If you haven’t got the knowledge to understand what it’s doing, haven’t got the discrimination to develop your own taste, and haven’t got your own uses in mind, then you might as well be using anything. A friend summarized (amusingly) a common attitude found on the internet regarding lenses:

I’ve just bought and paid for a new lens. Would you kind folks please offer your opinions on this lens so I can decide whether or not I should like it?

To paraphase a Leica lens designer (this may be apocryphal—I can’t find a source): "The only way to test a lens is to use it for a year. Everything else is a shortcut." I know I’m preaching to the converted here. But for those who perhaps haven’t worked all this out yet, stop fretting about lens tests and have confidence in yourself—look at pictures, and believe your eyes.

[Copyright 2001, 2009 by Michael C. Johnston. All Rights Reserved. First published October, 2001, in The 37th Frame.]

Comments

[Kevin McLoughlin]

I remember that article, Mike - along with another, from a little earlier, on lens sharpness.

Recalling the main points in both of them always manages to calm me down when I start obsessing about some lens or other that I probably can't afford and almost certainly don't need.

[William Kazak]

It is like growing roses. You have to own them to know them.Shoot in the conditions that you regularly find yourself in.Then, look at your results.Are you pleased?
I hate lens flare.It ruins the best pic of the series.When I discover that my lens is no good because it flares,then I sell it.After that,I see someone on a forum showing pics with that lens,albeit not the exact one that I sold,with marvelous example pics.It goes to show,there are sample variations and every lens looks good when the sun is not blasting.
The one year ownership rule makes a lot of sense.Even then,you might not realize all of the different f-stops in your tests, nor every different situation.Lenses are fun.I read lens reviews then I make up my mind if a lens is for me.I always stay with the manufacturer's offerings for quality.The exception?I bought a 500mm F6.3 Rokinon mirror lens for my Nikons.What a hoot at the beach!

[John Caruso]

Huzzah to you for this article. You've articulated many of the points I've long felt. Getting hung up in reviews, tests, and pages upon pages of internet praise or derision can make one's head spin. In the end, the proof is in the photo not in the pixels peeped or prints louped. Like you said, if the photos from a particular lens work, then it's a good lens. I learned a while ago that one has to trust one's own gut. I have lenses that others would not consider sexy or "good." However, I like what the do for me. I know what to expect from each of my lenses, and I know when I want to use them as well as for what purpose. I have also purchased lenses that I've returned simply because they didn't feel right. The photographs I made with them didn't sing in the way I wanted. Conversely, I've tried lenses of which I was skeptical...yet after seeing the results I knew I had keepers. In the end, it's all about the gut and the photos.

[Marcus]

There's a lot to understand when it comes to buying a lens. There are lot of factors to consider, but a professional will know what they are looking for, if they have done their research.

[Carsten]

Thanks for this great article!
Allow me to add that clenching experiences into number (8.5/10 pts. etc.) might not be very sensible, but it is the quickest way to formulate an impression a lens made. ;)

[multifocal lens]

The bird photo is really cool. very great sharing.