Not So Fast

It appears that yesterday, I was guilty of the common mental error of believing what I wanted to believe. I like the idea of Adobe DNG (digital negative)—a vendor-independent raw standard that works across platforms and promises unlimited future support. The DNGs I was shooting with the K20D were opening up in my older copy of Photoshop, and they looked great.

But appearances can be deceiving. Go into ACR (Adobe Camera Raw) and look at the numbers in the "White Balance" box, and you'll notice they can be way screwy—on my DNGs, temperature was 7000 Kelvin for one picture and tint was +97 for another, in fairly neutral light. That's not right. Carl tells me that's because the LUTs (lookup tables) are not proper for the camera and the raw file is not being unpacked properly. He wrote about this last year in his series of posts on the K10D, when he noticed the raw files didn't behave right until he got a version of ACR in which the K10D was supported.

So, even if you're using DNG, it's still necessary to use a version of ACR (or whatever raw converter you choose) that specifically supports the K20D (or whatever camera you own). With Adobe products, you need ACR 4.4 or Lightroom 1.4 to get K20D support. ACR 4.4.1, the latest version, is not compatible with versions of Photoshop prior to CS3, although it is usable with updated copies of Lightroom and with Photoshop Elements later than version 4.0 for Macintosh and 5.0 for Windows.

Carl is currently shooting PEFs (that's Pentax's proprietary raw file type) because they're smaller files, then immediately converting them to DNG using Adobe DNG Converter, which is a free download.

____________________

Mike  (Thanks to Carl for setting me straight)

Comments

[Shadzee]

I think most people who shoot RAW with K10D, use the same workflow;

Shoot in PEF >> Convert to DNG >> Lightroom/PS

I think this is the best workflow for Pentax. First of all the PEF files are much smaller (than the DNG produced in-camera by Pentax), and more efficient for your Memory Card use, and the DNGs converted by Adobe products are more "standards" compliant.

I use Lightroom, which has a great automated DNG conversion feature during the file uploads from the card.

[Carl Weese]

Shadzee wrote: "I use Lightroom, which has a great automated DNG conversion feature during the file uploads from the card."

For those not using Lightroom, you can use the Adobe DNG Converter to accomplish the same thing. Just launch the Converter, specify the card (either in the camera or a card reader) as the source and a folder on your hard drive as destination. The Converter will download/convert in a single operation.

In-camera K20D DNG files are about 24 megs. PEFs are about 17, either file type converted by the Adobe software ends up around 14 megs.

[Reinhard]

No, you can always use Adobe's free DNG converter to convert from PEF (or any other native format) to DNG; ACR and the free DNG converter are always updated at the same time. (-> http://www.adobe.com/products/dng/)

NB: In order to use these DNGs the Raw converter you use must support the DNG specification completely and not only support DNG for cameras it supports natively anyways. (This is no problem for all Adobe products, Silkypix and Capture One V4).

------
So, even if you're using DNG, it's still necessary to use a version of ACR (or whatever raw converter you choose) that specifically supports the K20D (or whatever camera you own). With Adobe products, you need ACR 4.4 or Lightroom 1.4 to get K20D support.

[Luke]

How are the PEFs much smaller?

On my K20D, when I set the RAW format to PEF, the camera says I've got space for 163 photos. Set to DNG, it says there's space for 162 photos.

I've used DNG since I turned on the K20D and on the K10D before that.

[Martin Doonan]

DNG seems to be going the way of pdf - apparently a common standard that actually has all kinds of features and support (from the latest software) to get it working properly. the one advantage that ought to come from DNG (that you can read unsupported camera files into older software) is obliterated.

[JohnBrewton]

Adobe now lets me know when they have an update available and I just click on their icon and it's done. No more downloads that can't seem to find my copy of PS anymore.

[Damon Schreiber]

For my K10D, I use Shadzee's workflow with the added twist of using Apple iPhoto to keep my photos organised. Importing is a bit of a long process, but I like the results.

I used to keep photos in iPhoto even after editing, but now if I want to do something with a photo more than admiring or emailing a quick copy to someone, I edit in ACR/Photoshop and then save separately. I tried using Lightroom and Aperture at one point, but they didn't seem to make me any more productive.

[Carl Weese]

Luke,

The counter reads (almost) the same because its software seems unable to deal with the compression. I consider this a bug. As you shoot down into a card, you shoot three and the counter goes down two. I did a lot of shooting with a K20 yesterday, with an 8-gig card that read 336 possible shots when I began. At the end of the shoot, it read 55 shots remaining, but when I went to download, there were 435 shots on the card. It would be nice if they fixed this.

[Peter Croft]

I'm using the Adobe DNG Converter v4.3 to convert Canon G9 .cr2 files, which still can't be read by most software, to .dng. But in doing so, I noticed a huge increase in file size, from about 12MB to 45-48MB.

Now, I've just bought a Ricoh GX100 which puts out native .dng raws of exactly 12.85MB. But if I run those through the Adobe DNG Converter, they end up the same 45-48MB size and noticeable cooler in tone.

It's not a real problem, but I don't understand it.

Pete

[john robison]

Am I missing something here? Or does the learning curve for digital seem vertical? It does whenever I read things like this last post by MJ! You know...'I tried to load the files into the CIA but FBI didn't read the FHA so the double cam , left handed festouris erased the NORAD and I had to reconfigure with a triple creme sunday application'

GWAD! I think I'll just get another couple hundred feet of Tri-X and some empty cassettes. :)

[Andreas]

Mike,

this weird white balance readings showed up only when I shot dng (or pef->dng with photo browser) with expanded d-range. And still I liked the colors more then from native PEFs, and couldn't really match them otherwise.

However, I think for the moment I won't use this d-range feature with raw.

Maybe you saw what you wanted to see, but I really hope that some day there will only be dng left in digital cameras. Everything else is proprietary and tedious.

Andreas

[Mike Johnston]

John,
I sympathize and apologize. That's one of the systemic problems of digital: when writing about technical aspects of film/darkroom work, the technical principles were usually broadly generalizable to other situations. With digital it's the opposite: every answer to every technical problem when answered exactingly is excruciatingly specific, and can't be assumed to be exportable to other, similar situations. C'est le nouveau monde.

Mike J.

[Morry Katz]

Right on John Robison.

If I wanted to play with software, I'd go back to school and take computer science.

I use a camera to take photos. On film! Then I make prints in my darkroom. Novel concept.

The tail seems to be wagging the dog.

[Martin Doonan]

"but I really hope that some day there will only be dng left in digital cameras."

I hope not. I do not want to be left in the hands of Adobe, thanks very much. Only hope is if DNG (or something similar) turns into a proper ISO standard.

BTW, looking at the file sizes reported, it doesn't seem to be much smaller than an uncompressed TIFF. Not exactly what you want from a camera file.

[thorir]

(at the risk of being thrown out of the party)

the key to the adobe dng is that it is an adobe format. and it won't be the universal format it's pretended to be until all manufacturers of digicams (slr and such included) use it, and use it in the same manner. only then we'll see raw processors reading and dealing with it properly.

as i understood it (and admittedly i don't have the tech knowhow to fully understand the issue), until recently no raw processors read the adobe generated dng's (some read dng's generated in camera, but only from cameras that were supported in that particular raw processor anyway).

not that the idea of an open standard raw format wasn't good, it was and is! but not if it means marrying to adobe for the whole workflow. as it is, dng is not the wonder solution it's presented to be...

[Eric Chan]

Hi Mike,

I'd like to clarify a few things ...

The white balance issue with the Pentax 20D's DNG files is actually a Pentax bug. The bug involves an incorrectly-computed color matrix being embedded into the DNG by the camera. It has nothing to do with Camera Raw. I believe Pentax is working on a fix.

With the fix in place, you should be able to open up the Pentax K20D DNG files with older versions of Camera Raw, even ones that were around before the K20D was released.

[John Robison]

Mike,
I still think this is a great site. You and the other contributing writers are as clear as the subject permits. Film photography is not really low tech when you consider the research that goes into the design and production of modern color negative or transparency films.
As far as regular darkroom work, I don't do it enough to be more than a hack printer. Inevitably, in any endeavor to produce fine prints, the best results usually go to those who work at it...alot.

Morris,
I love my film cameras but I am certainly not blind to the potential advantages of digital. I have a small tub of film I have yet to soup and the missus gets a little testy when she is trying to clean up our little apartment and I still have 20 strips of film hanging around not filed
collecting dust.

[Adam Maas]

The DNG issue is specific to the K20D and is most likely a bug in how the K20D records WB info into the DNG. Pentax will hopefully resolve this soon with a firmware update. The K10D and M8, being the other two DNG-native cameras, do not have this issue when used with older versions of ACR.

As to DNG becoming the standard, I'm not looking forward to that. I'd lose several of the capabilities of my Nikon D300 (like Auto CA removal and the D2X colour modes) if NIkon switched to DNG instead of NEF. While I appreciate the advantages of DNG in general, the extra capabilities that the NEF format offers when combined with Nikon's CaptureNX software more than make up for it.

[Charles Maclauchlan]

Although I applaud the concept of a standard digital negative, and I certainly feel the photographer owns the negative, not the camera/software company I still have to wonder how putting my eggs into Adobe's basket has done anything for me. True enough that Nikon doesn't control this process(for now) or Canon, or...whomever, but Adobe sure does.

[Andreas]

"BTW, looking at the file sizes reported, it doesn't seem to be much smaller than an uncompressed TIFF. Not exactly what you want from a camera file" (Martin Doonan)

Pardon, but an uncompressed TIFF with 14 mpix would be about 90 megs. This compared to a 15 megs PEF or DNG exeeds my tolerance for the usage of the word "much". While mathematically true, in common sense false.

And could anyone clarify: I thought that DNG would be an open standard, introduced by Adobe. Like TIFF, where I can download the specs, and if I am smart enough write my own TIFF application. That is not said I am smart enough, but there are people out there who are, like the Faststone guys for example.

Further I find it interesting that people vote for some real weird artifact like NEF and this itchy Nikon Capture which wouldn't run reasonably fast on my PC with D50 files 2 years ago while Lightroom still does with the K20D files now. And then they charge you extra money for that clunky piece of software, if you are not regarded a pro, i.e. you only buy a "consumer" slr. Bad luck for all that working pros which are fine with their D70s, D80s or whatever.

Thank you, but I think Adobe is one cool expample of a software company. I buy Lightroom once and don't have to worry about raw conversion from then on.

Andreas

[HepCat]

From what I understand, DNG is a lot like TIFF (and IFF for old Amiga heads)

They are all a wrapper and a description of how to interpret information that they contain. Obviously it is physically impossible to have a universal raw data format. If there are different designs of sensors, then they will produce different raw files. At the very least there are a couple different bayer sensor designs , the foveon , the various diagonally aligned Fuji designs , the above mentioned Fuji designs with two different sizes of all the sensors, the new Kodak patented designs with white pixels, and the Sony 828 sensor with two different colors of green. That just covers different physical layouts of sensor design, bit depth , non linearity , hot pixel mapping ( I assume there is hot pixel mapping in raw files , does anyone here know ? )

TIFF and DNG files have a header that describes what the data format is. That's why at least one raw format , Canon's is a perfectly legal TIFF standard file where the header describes the raw sensor data layout.

DNG converters are supposed to support DNG files of camera models of which they have no prior knowledge, but they can't support a camera with a new type of sensor.

There is of course "linear DNG" which aren't "raw" but are supposed to be able to contain all the possible information in a raw file.

In short , there is not now nor ( I hope ) will there ever be a "universal raw format"

The only way to have a "universal raw format" , is to freeze sensor technology, which would be a bad thing.

[01af]

As others already have mentioned, it obviously is better to shoot PEF and then convert those to DNG using the latest Adobe DNG Converter.

For those Pentax DNG images that you already have, converting them to Adobe DNG (using the Adobe DNG Converter) might be a good idea. At least, this was recommended for Leica DNG files out of the Leica DMR digital back.

By the way, did you notice that the DNG file format comes in three flavours? They are called DNG 1.0, DNG 1.1, and DNG 1.2. Adobe DNG Converter versions 2.3 through 3.2 created DNG 1.0 files; versions 3.3 through 4.3.1 created DNG 1.1 files, and finally versions 4.4 and 4.4.1 are now writing DNG 1.2 files. As far as I understand, all DNG format versions are forward and backward compatible ... so old ACR 2.4 can still read DNG 1.1 and 1.2 files (but cannot exploit the latest features of course).

-- Olaf

[PECourtejoie]

Peter Croft: What options are you using with the converter? it sounds like you are exporting to linear DNG, and/or maybe embedding the original file inside the DNG, making for a huge filesize.

I would not use a linear DNG with a Bayer pattern sensor, but rather with "exotic" sensors designs, and only to be able to open the files in some third party converters that do not support these sensors.

Pierre-Etienne

[Ade]

I have a similar problem with Bibble. If I save a raw scan from VueScan in DNG format, Bibble won't open it, even though it claims to support DNG. Why? Because the DNG is basically a wrapper around a plain TIFF file, and Bibble doesn't currently support native TIFF (even though many raw formats are actually variations on TIFF). Reading the small print, Bibble only supports DNG for certain supported cameras. Not so universal then.

[David Bostedo]

"I buy Lightroom once and don't have to worry about raw conversion from then on."

Andreas - That's only true as long as you have a camera that's supported, and a newer version of Lightroom is not out yet. Usually you can only upgrade ACR (the "Develop" engine in Lightroom) in the newest version of Adobe tools. For instance, if you have Photoshop CS2 and buy a K20D, you can't upgrade ACR to support it; You have to buy the newer Adobe software.

Unless they've changed this with Lightroom as opposed to Photoshop, you won't be able to keep upgrading forever...

[Eric Chan]

Adam Maas, there is no reason you would lose any of Capture NX's special abilities such as Auto CA correction, etc. if Nikon embraced DNG. The DNG file preserves all of the Nikon-specific metadata, even ones that aren't used by Camera Raw itself for image processing.

All that is required is for Capture NX to support the reading of DNG files (there is a public and free DNG SDK that would help them do this).

Same story for any other camera vendor (e.g., Canon).

[Peter Croft]

Aaaaah, Pierre-Etienne, thank you. I hadn't noticed the Linear setting - that's the default, I didn't set it that way. Yes, now that I've set it to Preserve Raw Image, the G9 conversions are slightly smaller, and if I set the jpg preview to full size, they are the same size as the cr2 files.

Merci beaucoup.
Peter

[Andreas]

http://www.digitalphotopro.com/tech/dpp-solutions-a-look-at-dng.html

Just confirms what I thought.

I know that the world isn't black and white but still can't resist this time:
Adobe rulez, Canikon sucks ;-))

Sorry and best always
Andreas