It's a fact of life for equipment reviewers that some readers have a greater comfort level with charts and graphs than with discussions of subjective trials. This is true even when the charts and graphs are merely graphic representations of subjective or holistic judgments, and not real data! That’s the principle, for example, behind Popular Photography & Imaging’s "SQF" charts. Although masquerading as data, complete with numbers in boxes carried to a tenth of a point, the charts are really just graphic representations of "subjective quality," which is what the "SQ" in SQF stands for. Pop has never, to my knowledge, related to its readers exactly how it relates hard data to these summary numbers.
Measurements, which might seem more trustworthy on the face of things, can make us overlook the obvious benefits of careful observation. In this thread on photo.net about Carl's and my review of the Pentax 35mm ƒ/2.8 DA Macro, there are a persistent number of belittling comments from a polite, well-mannered fellow named Paul Wilkins. (I'm being facetious—he's not actually terribly polite. At one point he calls me "an incompetent BS merchant," which very much wounds me—I fancy myself a very competent BS merchant, thank you, and I think any of my high school teachers or college professors would attest to that.) Anyway, this fellow grabs on to the fact that he sees serious "vignetting" in my sunset illustration in the review (the correct term is "falloff," but I know what he meant) and then he hangs on to it like a pit bull with a tug-toy.
Er, okay—I like vignetting (here, I mean vignetting). I habitually added it to darkroom prints via edge-burning, and I often add it to digital photographs in Photoshop, as in the photograph below (the darkening on the right is natural—that is, the light was like that—and I added some darkening on the left to balance it).
I admit that a little vignetting is unlikely to bother me, and it’s certainly possible for two photographers to have differing standards in regards to lens falloff. But the evidence of one sunset shot isn’t telling, for a simple reason—when you’re pointing the camera at the sunset, the sky really does get darker the farther away from the light source you get. So even if the lens is "vignetting" a little, that effect is also mimicking a natural phenomenon that fits the scene, and the resulting picture looks perfectly natural. Such a reading, of course, depends on an ability to see pictures as pictures, as opposed to seeing them as test shots. I doubt any reasonable user would object to the physical falloff of the 35mm DA Macro wide open, although I’m certainly open to letting every user decide for him- or herself. But one shot of a sunset not only isn’t proof, it isn't even evidence.
And here's the saddest thing about that. I've been doing this for so long that I knew before I even posted that illustration that some blowhard would crop up harrumphing about the terrible vignetting they could see in that shot. I've been fielding nitpicks for so long that I know where the nitpicks are going to be coming from before they even exist.
Looking at pictures that are meant to be looked at
It's germane to remember that observation is just as important a scientific principle as measurement. In photography, it especially makes sense, because the main purpose of our results, for the most part, is to be looked at. Observation allows us to concentrate on what’s really important in a picture. What we conclude from our observations will vary with each user, but that’s as it should be. We all respond to effects differently, and we all seek differing properties in differing degrees.
In any event, I have to say that the prejudices and peccadilloes of various interpreters do afford a high amusement value sometimes. Just after our review came out, another review was published on a different site that came to a different conclusion than we did. That one was chock-full of scientific-looking charts and graphs, so some readers used it as a handy means of dismissing everything Carl and I had said. Which is certainly their prerogative. The thing that made me smile, however, was that the "scientific" review started out with the following statement in the very first sentence: "...these lenses are assembled by hand thus resulting in a limited production volume." The only problem with that is that the Pentax "Limited" lenses, despite the name, have never been, and are not now, limited production items—Pentax uses the word "limited" to signify "deluxe." And, all camera lenses are assembled by hand—how else would they be assembled, by little teensy robots, like a miniature automobile assembly line?! But I guess it’s still a better review to some people, despite having this obvious factual howler at the outset, I suspect because it has those all-important charts 'n' graphs.
Someday, I swear, I'm going to try an experiment. I'm going to add totally fake charts and graphs to one of my reviews just to make diehard "objectivists" feel more comfortable with my conclusions. You know, I'll have a chart, for example, showing SPFF—subjective purple fringing factor—graphically, and the chart will assign the lens under test a rating of BM7.13, where "BM," unbeknownst to readers, will secretly stand for "bothers Mike."
Your botheration factor, still and all, might differ.
Featured Comment via tom bullard:
Featured Comment by Bill Pierce: "I don't know if they still do it this way, but years ago when I visited Leitz in Germany they did very specific optical bench testing to analyze specific optical aberrations for the designers. They also posted pictures for the workers to comment on. I asked which was the best way of evaluating performance. I was told if you were a photographer and had the time, best to look at pictures."