DxOmark.com

[PRESS RELEASE]

Introducing www.dxomark.com, the first digital camera benchmark based on RAW image quality

Created by leading image science company DxO Labs, this free online resource delivers in-depth, objective RAW-based image quality data to enable fair analysis and comparison of digital cameras

San Diego, California, USA, and Paris, France—Nov. 17th, 2008—At the Image Sensor 2008 conference in San Diego, DxO Labs today unveils www.dxomark.com, a new website delivering key objective metrics of sensor performance for a variety of cameras measured directly on the RAW image. Available as a free online resource, dxomark.com makes it possible for the first time to assess the intrinsic quality of a camera before the impact of any RAW conversion.

“There are many valuable resources reviewing the image quality of digital cameras, but none of them consider the actual RAW signal straight from the camera sensor,” explains Nicolas Touchard, Vice President of Marketing, DxO Labs Image Quality Evaluation business. “Demanding photographers who shoot in RAW should only care about the genuine quality of the RAW image, yet until now they have had to rely on measures based on converted RAW images, obviously biased by the processing applied to them, whether embedded or performed offline with a software RAW converter. Furthermore, as RAW converters evolve and improve, the latent potential of RAW images can only be gauged by analyzing the RAW images themselves, projecting the potential quality achievable with the ultimate RAW converter. This is why we believe that our approach will dramatically change the way photographers evaluate digital cameras.”

dxomark.com provides in-depth measurements of all the relevant characteristics of a sensor: actual ISO sensitivity (which generally differs from the value reported by the camera), signal-to-noise ratio (SNR), dynamic range, tonal range, color depth and sensitivity, metamerism, etc. dxomark.com already covers 50 popular cameras, including Digital Single-Lens Reflex (DSLRs) and high-end bridge cameras offering RAW image format. The site will be updated on a regular basis with new cameras....

READ ON

_____________________

Mike

Comments

[Clayton Lofgren]

These are specs that mean something. Note at the top end how much you need to spend for a 0.1% improvement. I hope you do try the D3. It would be very interesting to see if your opinion rates it as close to the Sony as these people do.

[Jerry Thirsty]

So, is this the Product Specifications thing taken to an absurd degree? What good is it to the buyer to evaluate cameras based on pre-raw-conversion performance when the buyer will never use it that way? It seems analogous to trying to compare b&w films without developing them.

[fork]

@Jerry

I only shoot raw, and I use the same software to process all my files. I would be much happier to see reviews that compare how good the sensors are, rather than how much smearing and sharpening is applied by the camera.

[Jason]

Interesting how this is portrayed as a metric that evaluates IQ, but then starts in with such factors as sensor size and MP count to come up wit it's own DXO ranking. Don't get me wrong, for the tech-nuts, thus seems like a logical extension of the pixel-peeping crowd. However, since the measurements are taken pre-raw conversion, I can't help but wonder what the benefit of it is. After all, at some pont we have to do the raw conversion to get to the print. And, in light of the "Oxford" post made here just a mere two days ago, the minutia of such machinations really shouldn't matter too much, because ultimately, isn't it the print quality that we should be after, and not some itangible IQ measurement take prior to any processing or raw conversion?

[Robert Chapman]

My first reaction to this site was negative - isn't it after all just the final image that's important? But after visiting it I thought it was quite worthwhile and hype-free. Then again, I'm a Nikon shooter and my cameras did really well...

[HKF]

Sony A900 is reported to have 10 frame per second...

BTW, how to quantity the DxOMark Sensor rating... 80.3 vs 78.9 vs 80.6?

[Robert Harshman]

I just spent about 1/2 hour going over DXOMarks for various cameras, and while interesting, not sure how much it will or should influence what you buy. So many other factors impact image quality. DXO Raw processing software is just one of many possible enhancements.

As a long time user of DXO Pro/Elite I get a big uplift from it in image quality sometimes from processing RAw files in DXO. However, sometimes I use ACR instead as it offers more control when I need it.

But, anyway, One should view the numbers carefully from DXO Mark as compare the difference between a Canon G10 and most any DSLR. Note how large the difference is verses the difference between most DSLR.

Yet, if you've not read it, read Luminous Landscapes review of the Canon G10. Michael's comparing the IQ to medium format digital backs and it comes out OK.

So, DXOMark is quite useful, but only a small part of any camera decision if at all.

Regards,

Robert

[fork]

I just looked at the results, and I'm very happy that they show very clearly what we already know : the only way to get significantly better IQ is to get a camera with a bigger sensor.

[Awake]

Another meaningless blah blah blah measurement.

a) The choice of Raw processor and 'developing' skill will greatly override any measurement provided by the dXo ratings. What good is a great sensor rating if I personally do a terrible job 'developing' the photo, or use some sub-par RAW processor?

b) What do the numbers mean? So one camera has a rating of 89.3 and another a rating of 75.7... so what... does that mean anything? What is the practical 'real life' difference between those numbers? Is there a visible difference?

c) Real world situations? These measurements may be fine and dandy at ISO 100, but how do cameras compare at ISO 1600? In my situation (in-theater photography) I need the best low light image, and I basically keep my cameras at ISO 1600. What do the dXO ratings say about that? Absolutely nothing.

d) What about all the other stuff that goes into a camera purchase, such as lens selections, cost, suitability for the task?

This is basically a useless scale that will attract chart readers that never really take any photos.

[Mike J.]

"I need the best low light image, and I basically keep my cameras at ISO 1600. What do the dXO ratings say about that? Absolutely nothing."

Awake,
Actually, DxOmark.com has a category just for light sensitivity. So just go by that.

There's even a visual approximation of noise charted against ISO. Go to your camera, then click on "tonal range." Then click on the slider on the right.

The site as a whole might not be the ultimate judge of image quality, but it does strike me as another useful tool.

Mike J.

[Robert Noble]

I'm sorry, but this strikes me as a really misguided project.

I know that DxO is trying get street cred for its testing capabilities (hence this great "free gift" they're giving to the photography community; they obviously want to be THE place photographers turn to when it comes to testing), but any attempts to "quantify quality" are doomed to fail ("Coming next week: We measure which chocolate tastes best by analyzing the cocoa-to-sugar ratio!").

I'm also not sure it won't backfire as a P.R. move. Some of the things being said about DxO by owners of cameras that didn't come out on top are not really printable in a family blog like this, and I doubt those photographers will be eager to fork over money to DxO anytime soon. I know I won't, and my ox (or is that spelled xO?) isn't even being gored.

But how can we be certain that this isn't a gigantic hoax concocted by Mike J. to reinforce his point in the "Comparisons Are Odious" post two posts down from this one?

[Ken]

Every so often someone creates something --a product, a site, a service-- that seems to respond to a "Wouldn't it be cool if..." remark. These DxO ratings seem to be a perfect example. "Wouldn't it be cool if someone built an objective normalized data base of raw camera imaging quality?"

Useful? Naw, not in any truly practical way. But it's a terrifically entertaining time-waster and will doubtlessly provide "substantiation" for countless camera arguments! (I'm sure that the long-suffering Nikon folks are popping their corks.)

Thank you DxO Labs!

[Archer]

I found it pretty interesting, at least as a bit of web programming. Dynamic range seems to be the main differentiator in their DxOMark: cameras with nearly identical performance in Tonal Range, but very different performance in Dynamic Range scored very differently, even though the sample image ends up being nearly identical.

With any luck, heavy use of this comparometer will further depress the price of the cameras that I want to own, as they are the ones that scored relatively poorly while (apparently) having nearly identical performance to the eye (up to ISO 800).

[Greg Smith]

Looks like lots of sniveling and whining going on...I personally think this site is a real plus in that it is quick, to the point and very well done...what you might do with the information is up to you, a personal problem, if you will....

[Jason]

I'm the first to say that it's what the user sees it what counts most, regardless of numbers.

When I first saw this, I had flashbacks of the worthless arguments on Usenet forums over Photodo.com lens test results.

I went to the site anyway, and as Mike said it's a tool - nothing more - but a useful one. After doing some tests it confirmed some suspicions I had about my camera and its high-ISO performance.

If anything, it reminded me to trust my eye even more as the results echo many of my own preferences. Yes, maybe we only see what we're looking for in results like these, but so far I find that the test results and my experience with the cameras seem to be in sync, and may have saved me a few hundred dollars in the process!

[Robin P]

Oh no! - another place on the web to waste time when we should be out shooting pics:)

Olympus owners won't be very happy...

Seriously though, I found it very interesting and would have been even more pleased if it contained data for older cameras so we could compare figures for models we used to own and love.

Cheers, Robin

[Mark Roberts]

You know, to *really* peg the irony meter you should have posted this immediately after the "Comparisons Are Odious" post!
;-)

[Kurt Friis Hansen]

It's interesting to see, that differences between same size sensors (i.e. APS-C) are rather "minimal".

I have used the EOS 350D extensively (approx. 35.000 exposures - the last 20-25.000 RAW only) and now use mainly the EOS 40D (approx. 15.000 RAW exposures); the DxO-mark seems to reflect my own experience very well.

I've handled a few other cameras (DSLR or not) briefly and the DxO-mark will probably provide a reasonable basis for judging sensor "RAW performance" (excuse the pun ;-).

[JWAddison]

Seems to me this is a useful tool. It's the final picture that counts, but aren't you better off starting with a better raw file? Isn't it analogous to trying to make a good print from the best negative possible?

[MarcinB]

Hover over dots on Tonal Range chart and you will notice how some brands cheat at high-iso game. For example, official (manufacturer) ISO 1600 was measured as ISO 1125 for Nikon D90 and ISO 1165 for Canon E450. On the hand, Olympus played fair, E520 measured at ISO 1702.

[Graydon]

The web site design strikes me as needing a swift kick; if you don't have flash turned on by default it panics.

I'm also mildly amused that the Samsung GX20 and the Pentax K20D get significantly different scores. Guess it's not the same camera innards after all....

[hugh crawford]

This is pretty interesting, sort of like finaly talking about film after years of talking about cameras.

Two things, I don't see what they use for a lens, presumably the put the cameras on an optical bench but it's odd they don't go into detail about it.

The other thing that would be really useful is if they would make the raw reference files available. The problem with all the test sites is that they don't isolate the camera sensor from the rest of the imaging chain.

[Christoph Hammann]

So, fine, now digigraphers have a single number to compare their cameras by: DxOMark Sensor. Sure to be much used in forums all over the internet. Doesn't help a bit during actual photography. Being a film shooter, I can't help but feel smug. If I don't like my current image quality, I change my film (size). Presto, new "DxOMark Sensor".

[Ctein]

Dear Folks,

I don't understand why this is provoking such controversy. It seems to me that what DxO is offering is something vaguely analogous to film technical data sheets.

Many, even most photographers have no need of such data sheets. For the ones who do, they are invaluable.

Understand that such data sheets don't tell you what YOUR prints will look like. They tell you about the film characteristics when it is processed in a standard way. If you process the film differently, you'll see different results.

Also, just because it's in the film doesn't mean you'll get it in a print. Any printer who's tried knows that it's an Olympian task moving the entire tonal range captured by a negative into a print (digital tools have reduced that to a merely Herculean effort). As a rule, you just didn't try-- you used what portion of the range worked well with your print papers and with your artistic intent.

None of that was controversial. A standard RAW characterization, if implemented well, is no different.

I've no opinion about DxO's implementation in particular, save that the dynamic range they report for the Nikon D200 (11.5 stops) is a very nice match to what DDB and I found in our tests (11+ stops). So, obviously they're correct in that! [self-centered grin]

pax / Ctein

[Rob]

This is supposed to be about image quality, but I see nothing about resolution or accutance or anything to do with perceived sharpness at varying print sizes.

It would also be good to know their working methodology for their judgments, or it all begins to seem amateurish and arbitrary.

[Mike J.]

Rob,
You've read the whole site? For instance here:

http://www.dxomark.com/index.php/eng/Technologies/Measurement-definitions ?

Just asking.

Mike J.

[Andreas]

Thanks for the last link Mike. I myself was too lazy to read that and wondered about differnces in resolution between cameras.

It would be interesting if they additionally (hear me: additionally) "apply" that data taking resolution into account. Say what numbers came out for a given print size. Because then the Sony A900 would be much better i guess.

Still I understand what this is about and find it ok. Furthermore it is very well implemented and is an endless source of time wasting joy, just like TOP ;-)) Sorry Mike, that must have been.

best always
Andreas

[Ctein]

Dear Hugh,

The characteristics these folks are measuring are little affected by the lens, if at all. So long as it's a halfway-decent lens to begin with, one can do internal data consistency checks to find out if the results are being distorted by the lens.

The two areas where lenses really matter are measuring sensitivity and measuring resolution. It's a simple task to determine if the lens apertures are accurate, and I didn't notice that resolution was one of the things that these folks are measuring.

If you read through the site as I did, I think you'll agree that these folks are more than minimally competent. I think it's pretty safe to assume that they understand these basics of testing.

As for providing the RAW files, what good would that do? These are special test images to be analyzed by their software tools. Readers wouldn't learn anything from looking at them (that they wouldn't have already learned from the posted results).

I've considered the question of providing RAW files to readers for my own tests. I decided against it. If the pixel peepers are less competent than me (the overwhelming majority of them), they won't learn anything from the files that I haven't told them already, and they may very well draw erroneous results. Why should I waste my time arguing with people who know less than I do, about data I understand better than them?

If they are more competent than me, they're entirely capable of running their own tests.

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[tim darach]

Hi Ctein
Re supplying raw data, from a scientific perspective there's a lot to be said for open access to data (you don't have to argue with them).

I guess that the issue with putting raw files online might be their size.

Regards, Tim

[Ctein]

Dear Tim,

Yeah, but this isn't peer-reviewed science. Emphasis on PEER.

I'm a reviewer and a teacher. When I'm dealing with my peers, I don't mind giving them access to original data. But, on the whole, my audience is (are?) not my peers.

pax / Ctein

[Bruno Trematore]

What I would really like to know is: how do I get 11 EV dynamic range out of my D200?

I tried photographing some gray charts and everything above +3.3 EV (more or less) gets clipped into white, and I worked with RAW files.

It might really be the case that the limitation is the raw converter (or the operator) and not the camera?

[Ctein]

Dear Bruno,

There are two things that you need to get right: one of them is the RAW conversion but the other and more important one is the original exposure. If you want to confidently photograph a scene with a very long luminance range, you need to spot-meter the extreme highlights and the extreme shadows and match those exposure values to the endpoints of the camera's capture range.

In other words, it's just the same thing you would have to do if you were photographing a very long luminance range scene on film-- automatic metering isn't usually going to cut it unless you're very lucky. Or you can bracket like mad; that's definitely more feasible with digital than with film. That's what I usually do, truth be told.

As for the RAW conversion, when DDB and I measured the 11 stop range at ISO 100 we just used ACR (version 4.5 at the time, I think) with the following settings:

Exposure-- 0; Recovery-- 30; Fill Light -- 0; Blacks -- 0; Rightness -- +50; Contrast -- +25.

Those, by the way, are close to my default settings for ordinary photography (normally my recovery value is around 20); I'd rather have a slightly flat image and kick things up in Photoshop than clip highlights or shadows before I have a chance to work with the photograph.

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================