DxOmark for 2011

This came from DxO's PR agency this morning—all a quote except where indicated—

DxO Labs is launching a new update to DxOMark, the free resource for comprehensive data produced by DxO Lab's testing of cameras and lenses, today. The site has been updated for 2011 with several new features, new reviews and even more summaries of the most interesting conclusions that can be drawn from this data. There are also new, easy-to-digest views and scores that put the headlines front and center.

[DxOMark 2011’s key features include:]

• A new ranking system for lenses that comes as a direct response to visitor’s feedback;
• Easier navigation and data access from Home to measurement pages;
• 200+ new lens/camera combinations—making a total of 1,000 available on the site!

Please visit www.dxomark.com to see all of information DxO Labs has made available to the public for free. As you likely know, DxOMark measurements are generated to improve the performance of products like DxO Optics Pro and rely on DxO Analyzer, the world’s leading turn-key laboratory solution for image quality evaluation. Thanks to its accuracy, completeness, and reliability, DxO Analyzer has become the reference tool for numerous leading imaging industry players.

-

Mike

Send this post to a friend

Please help support TOP by using our links to B&H Photo and Amazon! Many thanks.

Note: Links in this post may be to our affiliates; sales through affiliate links may benefit this site. More...
Original contents copyright 2010 by Michael C. Johnston and/or the bylined author. All Rights Reserved.

Featured Comment by Steve G, Mendocino: "Lens test results from DxOmark and other tech sites like SLRGear and Lenstip are a great resource and I'd never knock them—I've spent enough time in various labs to know how much work goes into their reports and how frustrating it is to put in those hours knowing full well the limitations and errors of the tools and methods therein.

"But they're only a start in evaluating a lens. The next step is, for me, the GBC (Girlfriend, Backyard & Cat) protocol—taking the darn thing out and taking a bunch of dumb throwaway pictures under a wide variety of apertures and conditions, perhaps even using the 'on a tripod' variation of the protocol, to find out how the darn thing works for you. That, not MTF or lines per inch, is what counts.

"Of course, some will insist on the WBD (Wife, Backyard & Dog) protocol. Time does not permit a discussion of the obvious technical shortcomings inherent in this approach."

Comments

[John Krumm]

I find their lens analysis not so great. They don't put enough effort into deciphering their numbers for understanding. Lenstip.com, a Polish site, does a much better job with lens reviews, I think (as does DPR).

[hugh crawford]

What I would dearly like to see in lens tests is a measure of the Axial Chromatic Aberration, and a measure of the resolution of the lens off axis where the image is in focus , not necessarily in a plane. I know that most lenses have curved fields, and that the curvature of field is often what is reflected in the MTF charts , but I'd really like to know how sharp the lens is at the edge when the edge is in focus rather than the center. I'd like to know how curved the field is. I'd like to know a lot more , but that would be a start.

[David]

Hi Mike
This will probably be responsible more more people changing brands by rating rather than looking at the pictures they make, I have never believed an MFT rating in my life preferring to look at the results.
On one forum there was great discussion about data bases which told the user what focal length on their zoom they were using, now I am sure Ansel, Robert and HCB will want one for their ipad.

David

[Max]

I see John made a comment on the lens numbers already. I think there are lots of quirks in just making math calculaions on lens and sensor combos, because lenses on smaller sensors look awful, while what you really get on paper at the sizes you expect to print when you use a m4/3 or an APSc camera looks excelent compared to a MF back or a full frame in their own sizes. For example I think the outcome of the Panasonic 20/1.7 is outstanding by any means, but, since its forcedly coupled with a half frame sensor it will always show as mediocre or worse if compared mathematically to any full frame lens on a full frame sensor.
In the way the lenses are compared I think it's just apples and oranges.

[Max]

I might not quite have nailed the issue on lens qualiy discrepancies, but just to present an example, the 18-55 Canon kit lens shows vastly better numbers than the Zuiko 35/3.5 digital macro, and that just doesn't sound right. You can also see the Zuiko lens changes punctuation according to body used. The whole thing doesn't sound quite right.
You could say "it's just the real results you get with each lens on each camera", I still don't buy it.

[Steve G, Mendocino]

Lens test results from DxOmark and other tech sites like SLRGear and Lenstip are a great resource and I'd never knock them - I've spent enough time in various labs to know how much work goes into their reports and how frustrating it is to put in those hours knowing full well the limitations and errors of the tools and methods therein.

But they're only a start in evaluating a lens. The next step is, for me, the GBC (Girlfriend, Backyard & Cat) protocol - taking the darn thing out and taking a bunch of dumb throwaway pictures under a wide variety of apertures and conditions, perhaps even using the 'on a tripod' variation of the protocol, to find out how the darn thing works for you. That, not MTF or lines per inch, is what counts.

Of course, some will insist on the WBD (Wife, Backyard & Dog) protocol. Time does not permit a discussion of the obvious technical shortcomings inherent in this approach.

[Zalman Stern]

I wish one of the review sites would measure the entrance pupil distance for each lens they review. (Under a variety of parameters for lenses where it varies, e.g. with focal length on a zoom.)

Yes, there are a few databases of this info floating around in the panoramic photography world, but they are quite incomplete and it seems a simple thing to measure when you are going through the effort of doing a full review anyway.

Of course the manufacturers could just publish the data too. But outside of a few exceptions like Leica, they don't seem to do so.

-Z-

[Michael Farrell]

Re: Steve Mendocino's comment:

Wouldn't one need to purchase the lens first before you begin the GBC protocol?

Will retailers refund a handled lens even after only a day or two?

[Kurt Shoens]

The dxomark ratings have saved me a fortune. I compare each dishy new camera that comes out to what I already have ... not that different ... and don't buy anything.

I purchased a copy of their software in gratitude.

[ctein]

Dear Max,

I don't follow your point about format. So long as a lens is tied to a specific format, how it performs with those formats is all that matters. Saying one lens is at a disadvantage compared to another because the format sizes are different is meaningless.

A 35mm Leica prime is at a format disadvantage compared to a 300mm Rodagon. In terms of abstract optical performance, the Leica's a run-away winner, but 8x10 film format trumps 35mm every time, when all is said and printed. So how is it in any way unfair to say that the Rodagon lens+camera combination is optically better than the Leica lens+camera combination?

It's what comes out on the screen and the print that matters, not what the optical bench says. That's why DxO (and pretty much everyone else) report results in terms of pixels or standard print size rather than lp/mm.

And, by the way, the number for lenses combined with small sensor cameras DON'T look awful at all. They may not be as good as the ones for bigger sensors but they're hardly awful. And sometimes they're as good or even better. Size isn't everything.

pax / Ctein

[David Ralph]

When DxO first posted its ratings of sensors (and I suppose associated processing chips in camera), I thought, golly -- something scientific and completely devoid of brand bias. That actually may be true. I had high hopes, and I suppose we should be grateful that the data are posted for free. But, their numbers mean precious little to me in the end. They do not explain what characteristics the score really comes from. What do they really mean in terms of ability to make a great image, or in what kind of light? Just a number to me. Rather like Consumers Union saying that one camera gets an 85, and another gets an 83 or 76. Mystery to me. So what? Worse for the new lens reports. Would the score have told me that my early Nikon 70-200mm VR would be outstanding on crop sensor but would suck in the corners full sensor bodies finally hit the market? Will it tell me which lens has great contrast and color rendition, or a curved field of focus that will concretely and noticeably affect its ability to put a scene fully in focus? If so, I am not bright enough to understand that. So far, I would rather hear a report from an actual human being who has taken the lens out, made images and a print or two.

Maybe the data will explain more in the future.