Charts 'n' Graphs

It's a fact of life for equipment reviewers that some readers have a greater comfort level with charts and graphs than with discussions of subjective trials. This is true even when the charts and graphs are merely graphic representations of subjective or holistic judgments, and not real data! That’s the principle, for example, behind Popular Photography & Imaging’s "SQF" charts. Although masquerading as data, complete with numbers in boxes carried to a tenth of a point, the charts are really just graphic representations of "subjective quality," which is what the "SQ" in SQF stands for. Pop has never, to my knowledge, related to its readers exactly how it relates hard data to these summary numbers.

Measurements, which might seem more trustworthy on the face of things, can make us overlook the obvious benefits of careful observation. In this thread on photo.net about Carl's and my review of the Pentax 35mm ƒ/2.8 DA Macro, there are a persistent number of belittling comments from a polite, well-mannered fellow named Paul Wilkins. (I'm being facetious—he's not actually terribly polite. At one point he calls me "an incompetent BS merchant," which very much wounds me—I fancy myself a very competent BS merchant, thank you, and I think any of my high school teachers or college professors would attest to that.) Anyway, this fellow grabs on to the fact that he sees serious "vignetting" in my sunset illustration in the review (the correct term is "falloff," but I know what he meant) and then he hangs on to it like a pit bull with a tug-toy.

This doesn't seem like serious lens falloff to me. Others are permitted to disagree.

Er, okay—I like vignetting (here, I mean vignetting). I habitually added it to darkroom prints via edge-burning, and I often add it to digital photographs in Photoshop, as in the photograph below (the darkening on the right is natural—that is, the light was like that—and I added some darkening on the left to balance it).

Catherine, Age 1

I admit that a little vignetting is unlikely to bother me, and it’s certainly possible for two photographers to have differing standards in regards to lens falloff. But the evidence of one sunset shot isn’t telling, for a simple reason—when you’re pointing the camera at the sunset, the sky really does get darker the farther away from the light source you get. So even if the lens is "vignetting" a little, that effect is also mimicking a natural phenomenon that fits the scene, and the resulting picture looks perfectly natural. Such a reading, of course, depends on an ability to see pictures as pictures, as opposed to seeing them as test shots. I doubt any reasonable user would object to the physical falloff of the 35mm DA Macro wide open, although I’m certainly open to letting every user decide for him- or herself. But one shot of a sunset not only isn’t proof, it isn't even evidence.

And here's the saddest thing about that. I've been doing this for so long that I knew before I even posted that illustration that some blowhard would crop up harrumphing about the terrible vignetting they could see in that shot. I've been fielding nitpicks for so long that I know where the nitpicks are going to be coming from before they even exist.

Looking at pictures that are meant to be looked at
It's germane to remember that observation is just as important a scientific principle as measurement. In photography, it especially makes sense, because the main purpose of our results, for the most part, is to be looked at. Observation allows us to concentrate on what’s really important in a picture. What we conclude from our observations will vary with each user, but that’s as it should be. We all respond to effects differently, and we all seek differing properties in differing degrees.

In any event, I have to say that the prejudices and peccadilloes of various interpreters do afford a high amusement value sometimes. Just after our review came out, another review was published on a different site that came to a different conclusion than we did. That one was chock-full of scientific-looking charts and graphs, so some readers used it as a handy means of dismissing everything Carl and I had said. Which is certainly their prerogative. The thing that made me smile, however, was that the "scientific" review started out with the following statement in the very first sentence: "...these lenses are assembled by hand thus resulting in a limited production volume." The only problem with that is that the Pentax "Limited" lenses, despite the name, have never been, and are not now, limited production items—Pentax uses the word "limited" to signify "deluxe." And, all camera lenses are assembled by hand—how else would they be assembled, by little teensy robots, like a miniature automobile assembly line?! But I guess it’s still a better review to some people, despite having this obvious factual howler at the outset, I suspect because it has those all-important charts 'n' graphs.

Someday, I swear, I'm going to try an experiment. I'm going to add totally fake charts and graphs to one of my reviews just to make diehard "objectivists" feel more comfortable with my conclusions. You know, I'll have a chart, for example, showing SPFF—subjective purple fringing factor—graphically, and the chart will assign the lens under test a rating of BM7.13, where "BM," unbeknownst to readers, will secretly stand for "bothers Mike."

Your botheration factor, still and all, might differ.

_____________________

Mike

Featured Comment via tom bullard:

Featured Comment by Bill Pierce: "I don't know if they still do it this way, but years ago when I visited Leitz in Germany they did very specific optical bench testing to analyze specific optical aberrations for the designers. They also posted pictures for the workers to comment on. I asked which was the best way of evaluating performance. I was told if you were a photographer and had the time, best to look at pictures."

Comments

[Jim]

Perhaps your recent vacation did its job, you relaxed and forgot the cares of the world ( in particular those of most who haunt the internet). Keep up the good writing, there are a few of us, actually probably quite a few, who are interested in what you have to say. And if we dont we just go out and make our own pictures, not rant on about others or what they've written.

[Janne]

Both your and Photozone's review are relevant. They don't contradict each other; they just don't look at the same things. Optical performance as measured by Photozone is very relevant for some uses (especially a macro like this that may see use in a lab setting - we have one in our lab already). The overall visual impression of the lens is of course enormously important for most uses the lens is going to be put to.

The story would not be complete without having both aspects of the lens covered. Your review and Photozone's complement each other very nicely.

[David]

Lovely delicate colours in the shot of Catherine.

:-)

[Dwight]

I had a classmate who would memorize the lines on graphs from his physiology book. He had no concept of what the lines meant, he just memorized them for the test. He passed the class and now he's a doctor. He treats patients in a large practice. Yikes!

[Adrian]

Talking about test charts, here's a document describing the mother(s) of all test charts:
http://ldcm.nasa.gov/library/HSRCIW01/IKONOS_VV_Overview_Pagnutti.pdf
And one of the real things in Google Maps:
http://maps.google.com/maps?f=q&hl=en&ll=30.515208,-86.308486&z=18

Yes, it was a boring sunday afternoon ;-)

[Mats Nilsson]

When the first Playstation console was released, critics claimed that it could not possibly perform the way it actually did (based on specifications). Nevermind that you could just pop in a disc in a machine and see for yourself. Sometimes, seeing is not believing.

[David Littlejohn]

I'll be reading along in one of your post or another, and there will be a picture like "Catherine, Age 1" and I'll sigh and think: Say what you want, the guy can take a picture and make a print.

[Trevor Small]

You stick to your guns, mate; you're dead right, photo's are for taking and looking at, not for measuring.

[Stephen Best]

Your review and that on photozone.de aren't inconsistent. The lens has good technical performance across the (limited) usable aperture range and for its weight looks well worthy of consideration. You do yourself a disservice by bagging Klaus' efforts.

[dale moreau]

I could go into one of my long rants comparing audio tests to camera tests because like most males in their 50s I had to have both to the extreme. Yes Steinway's and Kawai's both have 88 keys, and similar dimensions. why do they sound different? guess we need more tests to quantify the difference between a Rubenstein Steinway and a Horowitz Steinway, they were both convinced of the difference.
But my favorite story relevant to the post was an anecdote of Richard Feynman, a noteworthy Physicist. It was told by his companion an a trip to Cern. They blundered into an area where an impressive experiment was taking place and when Professor Feynman asked about it he was told that it was an experiment to try to verify some of his actual work. Feynman then asked "How much did this cost?" The technician replied "$15 million." Feynman then asked... "You couldn't just believe me?"

dale

[Mike J.]

Stephen,
I'm not "bagging" Klaus's efforts. It's a perfectly good review. Just making the point that relevance and accuracy don't necessarily derive from how information is presented.

Mike J.

[Hoainam]

Ha ha ha: BM7.13... They should put you in charge of the Department of Homeland Security's Terror Threat Level too.

[Ed Richards]

Are lenses really assembled by hand? That would surprise me, almost everything complicated is assembled by tiny robots these days. (Actually, huge robots with tiny articulators.)

[Gingerbaker]

Well, I don't know about all you muscleheads out there, but I find pushing a mouse button hard work.

It has got to take at least two clicks of mouse to remove vignetting from a digital camera image. I'd add more here but my arms are about to drop clear off.

Peace.

[Gordon Buck]

So you *admit* to Photoshopping the picture of Catherine?

(What a shame that the typen word cannot convey the humour and sarcasm intended.)

A beautiful photo!

[Mike J.]

Ed,
There are a number of good videos online about lens production. Go to YouTube and search "How Optical Lenses are made" for one--I think it comes from the Discovery Channel. Others might have additional suggestions but that's a good place to start.

Mike J.

[Graham Hughes]

The Discovery Channel production is, I think, typical of a more small scale operation; as for the larger scale stuff Canon produced a mini-documentary on the process of making their 500mm lenses. It is highly mechanized and still incorporates a great deal of hand work to assemble the thing.

[rwzeitgeist]

The Canon "Virtual Lens Plant" at http://www.canon.com/camera-museum/tech/l_plant/f_index.html contains a wonderful video of a person assembling an EF 500mm F4L I USM lends. By hand, with the help of a lot of specialized tooling.

-Bob

[stephen connor]

Mike, you mentioned in an earlier post that you don't drink. Having read the posts regarding your review of the Pentax 35 macro from, um, what was his name again?, I have to say that you're a much, much stronger man than me.

[Dave Kee]

Too bad "what was his name again?" comes across as such an a** h*** as I like his photography. Of course, there is no connection between craft and personality as is evidenced by the fact that people tell me I have a nice personality.

[Bülent]

As an amateur photographer, I read those technical writings avidly, to satisfy my curiosity most of the time. It is like knowing the distance from earth to sun; it wouldn't help me in any way and I still like knowing about it. However, I "value" comments like yours a lot more when I am on the verge of a decision that could effect -however mildly- my life...

[Dierk]

Yeah, well, pseudo-science.

AS is pointed out by Scott Adams [over and over again] people love anything with numbers and graphs - where do you think self-proclaimed economic experts got their love for Powerpoint from? - as it looks mathematical and scientific. Since most dumbwits believe [yes, as in in 'religiously'] science to be about surefire facts, not about doubts and scepticism, anything looking remotely as science must be right. Hence Christian Science and astrology.

Isn't it interesting that those claiming decimal-point accuracy don't even get their basic natural and mathematical laws right? As you, Mike, point out, the light fall-off seen in the sky image [and every sky image] shows exactly what happens with light *in the sky*. It might be enhanced by the fact that the same happens quite naturally in the lens, which is buzilt to concentrate light towards the centre.

Ah, well, most people never understood why a CRT got deeper when the front got wider and flatter ...

[Erez]

First, three lovely pictures.

I love your passionate arguments about equipment, it's always refreshing to read. I'm yet to see anyone making good photo *because* of a lens, any lens. IMHO, photography is the art of seeing and then comes the rest.

Last, I just learned that lenses are made by hand. This makes me much more understanding about the cost.

[Jay Moynihan]

"Are lenses really assembled by hand? That would surprise me, almost everything complicated is assembled by tiny robots these days. (Actually, huge robots with tiny articulators.)"

Does Pentax say "human hand", or just "hand".

:)

BTW: Photozone.de has a review now under the Pentax section.

[Robin Parmar]

I have defended your point of view on photo.net and Pentax Forums, backing this up with actual photos taken with the lens. It's amazing how few supposed photographers do that. (Granted, it takes time.)

I am also interested in the Photozone tests, but do not by any means take them as gospel. Not every characteristic of a lens can be neatly summed up in figures.

Your photo of Catherine is lovely. It is equally true that for other applications any light fall-off is detrimental -- this is a macro lens after all. My suggestion to photographers using the DA35 Limited for those tasks is simple: stop down.

[J_G]

I never thought much of MTF and all other charts. Oh, wait, just once: Remember how they "proved" that Takumars were as good as Leica lenses?

To me vignetting, resolution, sharpness, etc characterise a tool. I may need to go one route or the opposite, using a Diana or a pentax 6x7. Reviewers who discuss numbers and assume there is only one stick with wich to measure how good something is are preachers of a very strict religion. Not interesting, except for acolytes of the same religion.

What I appreciate in your case (as in all the critics and reviewers I trust) is a consistency in your judgments through time. So that if you like something, I'm pretty sure I am going to like it too. It works also in reverse for some other reviewers There are a couple of film critics whose recommendations I do follow: If they don't like a film, I am pretty sure I'll enjoy it.

[Benjamin Marks]

Well yesterday I had a camera come back from a several months'-long loan to a friend. I wanted to see how the 1960's era lens on that camera compared to a more recent lenses, when used wide-open. So out came the ruler and the tripod. Here are the results:

50mm f:2 Planar
50mm f:3.5 Heliar-S w/hacked Contax to Leica M-adapter
35mm f:2 Aspherical this-n-that

These lenses were all _freakin'_ sharp. I mean they were so freakin' sharp that you could count the zits on the posterior of the dust mites camped out between the millimeter marks on my test-ruler. SHARP.

50 f:2 Summicron (from the 60's)
50 f:1.5 Sonnar-C (some focus shift stopped down)
50 f:2 Heliar (interesting abberations wide open)
40 f:2 Summicron

These lenses were just plain sharp. I mean they were slicing photons with reckless, but ordinary, abandon. You could see the dust mites, but not the gang signs they were flashing.

But then I started thinking about how to approach my mother (in her 70's) about a portrait, and about how her favorite lens of mine is actually an uncoated 19th century brass barrel lens with no shutter and some unidentified gunk on one of the internal elements (wait . . . is that fungus?). And then I started thinking about the last time I actually needed the millimeter markings on a ruler separated by day-light in my photos. And then I got all sentimental and weepy about the thought of using a lens that was freakin' sharp on woman who has seen her share of troubles with a wayward, photo-gear obsessed son -- I mean what had she done to deserve that? And I reached for the Heliar. Now it might be freakin' sharp when stopped down, but there was definitely some glow-y, abberational, pixie-dust being scattered liberally about when the lens was wide open. And sure, the MTF graph for this lens at f:2 looks like the micro-encephalograph of a lab-rat being used to test the effects of Lucky Charms on 4 year-olds -- but hey. This is Mom, we're talking about.

And so, my fellow Americans with apple pie in my heart, in these troubled times I say: remember Mother before you reach for those graphs and charts. Or the keyboard.

Nice snaps, Michael.

Tongue firmly in cheek,

Ben Marks

Ben

[Lars K. Christensen]

The discussion on photo.net makes me sad. Mike and Carls review might be "wordy", but I appreciate it, not just as a review but more as an an oppiniated essay on what defines a "good" lens...
On the other hand, I think Paul Wilkins has a valid point. Actually, he is not saying that this lens is ruined by vignetting - just that its there, and that this fact has been understated by Mike and Carl.
So far so good. The sad thing is, that a some point there after, both parties seem to deliberately misunderstand and exxagerate the point of the other. This turns what could have been an interesting exhange of opinion into simpel mud-slinging.

[Daniel]

I do dimly recall that PopPhoto did have an article ages ago where they explain the basis behind the SQF, I think it was when they first started to use it in their lens reviews. There is short explanation by PopPhoto here http://www.popphoto.com/cameralenses/2564/optic-nerves-afraid-to-buy-the-wrong-lens-sqf-subjective-quality-factor-page4.html

As I understand it, SQF basically is an integral of the MTF over visually crucial spatial frequencies, these being determined by the assumed print size/viewing distance of the viewer. There have experiments to see how well various image quality metrics correlate with the judgements by viewing panels. Some online references are http://www.imatest.com/docs/sqf.html
and http://www.bobatkins.com/photography/technical/mtf/mtf1.html

The original paper describing SQF is "An optical merit function (SQF), which correlates with subjective image judgments", E. M. (Ed) Granger and K. N. Cupery of Eastman Kodak, published in Photographic Science and Engineering, Vol. 16, no. 3, May-June 1973, pp. 221-230

Cheers,
Daniel.

[Miserere]

I'm saddened that "my thread" ended the way it did. I started it out of admiration for Mike, and as a way to alert fellow Pentaxians on the forum of Mike's article (which is not that prominent and might go unnoticed if you're not looking for it).

I didn't expect everyone to agree with their conclusions, but I did hope for some civility.

The worst part is that Mike and Paul actually agree with each other, but for some reason you guys latched onto the differences between you and it all went downhill from there.

Paul was brusque with his comments, and then you, Mike, got annoyed by him. I understand. The rest is p-net history. And now I'm left feeling like I should call the both of you into my office and ask you to shake hands. Because the thing is, you both *like* this lens! Paul uses it a lot and I've seen his pictures; he wouldn't have taken them if he didn't like it. And I've heard you Mike, going on about this lens for weeks, which you don't normally do, so I know you love it too.

Now, if I may scold you publicly, Mike, I'm saddened that you have decided that because of this incident you will no longer review lenses for your p-net articles. It sounds to me like you've gone into a vignetted huff more characteristic of cute Catherine than of someone with your experience.

As a scientist, I appreciate numbers and graphs, but I can tell the difference between Art and Science and I think I'm pretty good about knowing when to apply one and not the other. You are too, which is why I follow your writings and trust your opinions.

So, Mike, I publicly get on my knees and kindly, but firmly, request that you reconsider your decision and publish at least half of the lens articles you had initially planned.

I can assure you that the number of people who appreciate your articles far outnumber those who don't.

Best,

--M.

[Eolake Stobblehouse]

Even if the sun is not the middle, you usually get vignetting when photographing into water with a wide lens, because the difference in view angles make the surface reflect differently.

[Mike J.]

Miserere,
I commiserate. And I thank you for opening the topic; your intentions were the best and your enthusiasm was obvious. Sorry it came down to a petty spat.

Mike J.

[xtoph]

i just saw canon's web page for the new 50d, and i thought it was kind of amusing that the photo they used to illustrate their new "peripheral illumination correction" feature... is a picture of a sunset over water.

see here: http://web.canon.jp/imaging/eosd/eos50d/01.html#02

maybe they read t.o.p. too. ;)

[John Banister]

Saw this chart & thought of you...

http://www.theonion.com/content/files/images/soda-can-wealth.jpg

[Michael Barkowski]

This is basically why I don't participate in certain genres of forums so much anymore. Somehow, they have the same effect as putting all the participants in a hot, crowded room; sooner or later, someone throws an elbow, and people start to embarass themselves.

Any tips you might have on getting a 1-year-old to stand still and look at you for a picture, send them my way. My daughter is 19 months.

Cheers,

Michael B

[Mike J.]

Michael B,
Best strategy at your daughter's current age is to get some other adult to hold her. ("Ride the horsey" position, on a knee, facing outwards, works well, and the other adult can hold her around the waist and not have her hands in the picture.) In another year or two, try this: build a sturdy, low, fairly narrow cube of plywood. Set your lighting for and pre-focus on where the box is placed. Then ask her if she thinks she can stand up on it. What she'll do is step up on it hesitatingly, get both feet securely in place, realize she has had a SUCCESS, and then, briefly, she will look up at you with a radiant, delighted smile.

You get one shot; be ready. [g]

Mike J. (who's used that trick too many times to count)