The Storage Dilemma

I've always welcomed comments on this site, and sometimes the comments, collectively, are better than the original post, but I seldom simply toss up a subject and open the floor to other speakers, depending on commenters to provide the information and interest. But I think I'll try that now. The occasion is a question from an "old pro," Scott S., that I can't answer:

I read your site regularly and enjoy it immensely. I have been a pro for 25 years, working mostly for non-profits.

I am faced with a storage issue. I have strung together pairs of hard drives (6 sets totaling about 7 TB to date) that serve well for short-term storage. What do most people do for a long-term solution? I have heard so many options, I am just confused. Do you know a site or link that may help?

What about this? What does the future of archiving look like for digital images? The archiving of prints, slides, negatives, and other traditional media, a sub-specialty of paper conservation, has a well-established science and standardized practices. But what can you do now, or what do we need to achieve in the near future, for long-term, permanent archiving of our digital media?

Does anyone know what actual archiving institutions are doing? Is anyone aware of the state of standard practices among conservators?

______________________

Mike

Featured Comment by Craig Norris: "This subject has been actively on my agenda for the past ten years, ever since it became my responsibility to work on the problem for TV stations (my other career outside photography). Video production typically consumes up to 70 gigabytes for every hour of high definition video that is recorded in some form of compressed file. Even with low-end video formats like HDV, using much deeper degrees of compression, it still requires something in the order of 11GB for each hour of recorded material.

"In still photography the size of our problem is some order of magnitudes smaller in terms of absolute data storage consumed, but the requirement to faithfully preserve the data indefinitely is no less important and no less difficult.

"To get straight to the question posed by Scott, I can pass on the fact that TV stations today are already using or looking to use LTO data tapes as the main digital bulk storage medium. The current version called LTO-4 has a native capacity of 800GB per tape cartridge.

"Linear data tape cartridges aren't good for day-to-day random access, so the common practice is to keep the active material on RAID hard disk arrays, and to back it up regularly to LTO-4 tape, which is usually housed in robotic auto-changing libraries containing hundreds or even thousands of tape cartridges. Next year will see the introduction of LTO-5 tape, with double the capacity of LTO-4.

"The above-mentioned solution being adopted by TV stations isn't economically viable for the average photographer, except for the fact that a standalone LTO-4 tape drive can be used to back-up the multi-terabyte hard disk storage attached to one's workstation.

"According to my latest research on behalf of TV stations, and in the quest to solve my own data storage problem at my studio, blu-ray disc is emerging as a practical data storage medium. With the new dual-layer version offering 50GB of capacity per disc, and an estimated data shelf life in the order of twenty years, blu-ray disc is likely to be the next data archival medium that I implement for my digital photo archives.

"Supporting my comments above is the observation I made a few days ago when I was hired to photograph the NBC Universal Olympics facilities in Beijing. I saw NBC recording a very large proportion of the Games broadcast on Sony's XDCAM HD format discs, a close cousin of the Blu-ray data disc."

Featured Comment by Stephen Barnes: "Information Management is my day job, and the one thing I'll say is that it's just as confusing for the professional archivists as it is for the photographers.

"I've worked with the National Archives for Scotland, Public Records Office for Northern Ireland, and The National Archives in England, and none of them have any consistent method or approach.

"The first problem is archive longevity. Any images we needed to send to them were put onto MAM-E [MAM-A in America] Gold DVD disks which have an estimated 200 year retention period. I personally use Kodak Professional Archival DVDs. All images are then copied onto a SAN which retains multiple copies of each file on-line. Finally, this SAN is backed up to tape. We therefore have three different technologies on which images are retained.

"If a technology looks like it is going to be superceded, we transfer all data to the new medium as soon as possible, to make sure we have the ability to read the data. Old technologies (eg. floppy disks, even VHS!) are archived along with a number of working devices. (There's no point in archiving something if you don't know if you'll have a device in the future!)

"The same is true for formats. While it doesn't look like JPG/TIFF are going to go away we would convert to alternative formats (PNG, PDF, DNG etc) to minimise risk.

"What this all means is that for sensitive and historically important information we have up to four copies of the same data on each of three media. That's 12 copies of our images.

"But the professional archivists have a preferred media for long term preservation—silver halide prints."

Featured Comment by David A. Goldfarb: "If one wants to preserve images for the ages and not rely on data transfer, I agree that the solution is to make the best archival prints one can make and store them according to the accepted archival practices for paper media. An important principle of archiving is that a long-term archive should be 'human readable' if possible.

"I met a historian once who had responsibility for the archive of an important figure in U.S. foreign policy. He had an office full of 8" floppies, the original computer used to produce them, and a huge IBM printer the size of a coffee table that could print files from this computer. Sometimes that's what it takes to preserve something worth preserving in digital form. I suspect by now, if he hasn't migrated the data digitally, he's probably printed it all, and scanned and/or microfilmed it."

Featured Comment by Charlie H: "Personally, I am all about the photo-album. Last night we pulled out a few clunker sized albums of my wife's trip to China twenty years ago. Pics looked great, the kids were all over it. We haven't so much as lifted a finger preserving this archive. Barring a fire, they should be around for a good while and that's enough for me. Long live the photo album."

Mike replies: Charlie, you stole my thunder. I'm writing a column now making the case for the lowly album as a surprisingly good alternative for some people and some purposes. It's interesting to say the least. And it feeds into what Gordon says below, too.

Featured Comment by Gordon Lewis: "I don't mean this to sound gloomy, but if you look at the history of the world as we know it, nothing lasts forever, at least not in its original form. The best we can hope for is that our work outlasts us. Once we're gone and it's in someone else's hands, who knows? They may value it and want to preserve it; then again, they may not—and this assumes they know where it is and want to go to the trouble of examining it.

"A previous poster mentioned that there are short-term and long-term dimensions to storage. I would add that there are commercial and non-commercial dimensions as well. You can't sell what you can't find or can't read, so there's a strong financial imperative to preserving your work. Family and personal photos, though non-commercial in intent, could be invaluable to the owner, yet it's amazing how little time, money and effort the average person puts into preserving them.

"For my own part, I've decided that instead of simply preserving the photos I've taken of family and friends, I'm going to print them and distribute them as albums; to spread the wealth, so to speak. It's possible they'll last longer or at least be better appreciated if they're distributed rather than hoarded.

"I'm not lecturing anyone else, mind you; just sharing a few reflections. The way I'd summarize my feeling is this: Save your work as best you can, but also remember who you're saving it for and make them a part of the process; otherwise, your preservation efforts may end when you do."

Comments

[Mike J.]

The one contribution I'll make is that I am firmly of the opinion that any solution that depends on regular maintenance (the old "just transfer your files whenever a new storage medium comes along") is just not adequate. First of all, people just won't do it--if you're pressed for time and money, you're just not going to devote time and money to old files. Second, we'll soon simply have too much to do it with--7TB might be manageable, but as old material accumulates, archives become unmanageable in size. And, third, don't forget that at some point, archives get separated from their "husbander" and have to make their own way in the world. It's all well and good to say "just transfer the files to a new medium every ten years," but what happens after you die and there's no one who knows the files need transferring? Plenty of old prints and negatives have survived "on their own," as it were, for decades, until being rescued or rediscovered. Are we just going to give up on that possibility where digital files are concerned?

Mike J.

[psu_13]

Digital files survive if you make a copy of them every once in a while (a year or two, I expect). Therefore the best short to mid-term solution is to buy new disks every year or two and make a complete copy of the entire archive.

I'm not sure what you do beyond this because I don't really trust any of the non-disk media (e.g. optical disks, tapes, etc). Certainly making prints of the best and most important pictures is a good policy, but current color prints don't really last all that much longer than a DVD.

Maybe someone will come up with a way to transfer digital files back to film. :)

[Edward Taylor]

Long term storage is a problem (as it always has been).
I could write a book on this, but the short answer is that there is no definitive answer to this question for two main reasons. One is that no one knows the media that will work best or last the longest, and with technology always changing, both the media and the software to read it will constantly be changing. Right now, Blu Ray may be the best storage media (multiple copies of everything still) and a standard file format like DNG, especially for RAW files. I suspect we will all need to constantly convert and transfer images from one format to another and from one media to another if we want to save them for hundreds of years.

Storing photos as prints is too limiting in light of all the options available now and in the future for viewing and distribution, etc.

[Ben Curtis]

First off, there is NO ideal solution that I can see. And it sucks. We now acquire images at an alarming rate, documenting events and memories whether public or personal, and there is just no good, cheap, reliable way of storing them.

All media deteriorates over time and runs the risk of getting damaged, stolen, burned, or just corrupted.

You say you disagree with the "just transfer your files whenever a new storage medium comes along" method. Maybe, but realistically I don't think you really have any choice - you WILL have to at some point. Personally I accept and welcome that I will have to transfer them at some stage, and that is why I prefer multiple hard drives as a storage medium.
1TB of photos on a hard drive can be transferred (with verification) to new and bigger drives in less than a day. Try doing that with 1TB worth of CDs or DVDs (approx 1400 CDs or 200DVDs).

In an ideal world we'd have reliable, long-life, optical disks of 1TB capacity priced at $10 each. It will surely come but we're not there yet and so far optical media has not kept up with the capacities of images we're shooting.

Currently, Blu-ray is probably the closest, but at current pricing per 25GB disk it's no cheaper than buying hard drives as a single-use medium.
And unless you store optical disks under good conditions, do not expect them to last forever.

My archive isn't up to 7TB, if it was maybe I'd think differently, but so far I'm sticking with RAIDed hard drive arrays.

I wrote a couple of blog posts on this subject a while back, you might find them useful:

http://www.snappertalk.com/index.php/20080604/archiving-photos/

http://www.snappertalk.com/index.php/20060309/raid-photographers/

and these on another website who seem to have quite a well thought out system:

http://www.deathtofilm.com/2006/11/12/image-mechanics-off-line-archiving-system/

http://www.deathtofilm.com/2008/06/30/image-mechanics-off-line-archiving-system-the-next-generation/

But like I said above, in my mind there is currently no great solution to the widespread problem of how the hell do we store it all and not lose anything...

Cheers,

Ben

[OnStage]

I don't have the answers either, but I can add two important points:
Use an open standard, I am migrating to .dng as it matures but .jpeg or TIFF would also work if you don't need to archive RAW.
When I am creating my long term off-site storage I include not only the files but everything needed to read them. That means cables, power supplies, image software, OS, and will include a retired computer as soon as I can afford the luxury!
As Mike says; prints are still your best bet, especially considering that they reflect not only the file as it was captured but also the image as you imagined and proccessed it.
cheers,
john

[Edward Taylor]

Mike,
If one considers the restrictions in your comment, then the answer is that right now there is no solution. To go with an audio analogy--in the past 15 years, I took the time to transfer many of my tapes, minidiscs and records to DAT. Now I cannot buy a DAT player. Try to find a Betamax player or an eight track. In a short time, many audio formats have become obsolete. I have Laserdiscs, SACD's, DVD-A's, etc.

There are no long-term standards and I suspect there won't be for some time. Eventually, the limits of human hearing and sight may make it reasonable to stop trying to improve on sound or video, but we will never stop trying to make it smaller or more convenient. That's how MP3's took off. I assume at some point in the future, I won't be able to find anything that will play my MP3's. Windows 28 will not support any program that can read an MP3.

[Jeff M]

My long term solution is this: I make a really good archival prints of images that I care about and store them in archival boxes. I have been doing photography long enough ( 25 years ) to know that I really will not care about the vast majority of my out-takes years from now. Sure.. I have backups on hard drives of everything, but if at some point they are gone, my prints will satisfy 99% of my desires.

[improbable]

Another vote for multiple hard drives. USB hard drives containing jpeg files are likely to be accessible for a good while, I think. DVDs too perhaps but keep the main copies on disk.

I really think we're to some extent over the hump with regards to formats changing, 80s audio gear and computers changed every few years, but things have settled down now. The web forces everyone to keep supporting jpeg. USB will be upgraded but not thrown out, I think. Raw processors seem to be adding support but keeping all past cameras. I could be wrong but that's my feeling about the way things are going.

MikeJ said:

"The one contribution I'll make is that I am firmly of the opinion that any solution that depends on regular maintenance (the old "just transfer your files whenever a new storage medium comes along") is just not adequate. First of all, people just won't do it--if you're pressed for time and money, you're just not going to devote time and money to old files. Second, we'll soon simply have too much to do it with--7TB might be manageable, but as old material accumulates, archives become unmanageable in size."

This has been, at least for me, not an issue. Each new camera's files are bigger, and its contemporary hard drives are bigger too, so that keeping all previous years' stuff next to this year's isn't a big headache.

If data grows by Moore's law, i.e. exponentially, then all past years together are some fixed number of times bigger than this year.

[stephan mantler]

On a broader scale, there's a EU network for long-term archival (see http://www.langzeitarchivierung.de/index.php?newlang=eng ), and they are organizing a lecture at the photokina 2008. So if you're in Cologne, this may be worth attending...

In short, it appears that libraries and museums have already begun to print digital images to microfilm in a way that can be redigitalized losslessly.

A more detailed programme (in German) can be found at http://www.dgph.de/sektionen/wissenschaft_technik/symposium08.html#programm

I've copied the news release here:
~~~~
DGPh-Lecture forum "Preserving digitally, preserving on film?" ("Digital speichern - filmbasiert archivieren?") at the photokina 2008, Cologne

The section science and technology of the German Society for Photography (DGPh) organizes together with the Cologne fair and Prophoto GmbH at the photokina 2008 a lecture forum “Preserve digitally, preserve on film” which reflects the possibilities and limits of either film-based or digital preservation.
The important problem of long-term preservation of digital images remains so far to a large extent unresolved: If the data of valuable digitally shot pictures is not regularly backed up on the next technological level - keyword „data migration “- important photographs can be lost. Films however – depending on appropriate processing and keeping - are steady over long periods.
The lecture forum takes place in the afternoon of 25th September and in the morning of 26th September in the crystal hall “Kristallsaal” of the Congress Centrum East at the Cologne fair.
The lectures are held by experts from different enterprises and institutions, followed by a final panel discussion, which tries to clarify, which problems film-based preservation can solve.
No participation fee. Informal registration appreciated: Deutsche Gesellschaft für Photographie, Geschäftsstelle, Rheingasse 8-12, 50676 Köln, Tel. 0221/9232069, Fax 0221/9232070, E-mail: [email protected], Internet www.dgph.de.
~~~~

[Derek]

A while back I heard an online talk by Magnum photographer Martin Parr. He said they back up their photos on gold-surfaced CD's, and also make a physical print of every photo. He added that they don't do it themselves, they pay someone else to go through the trouble of archiving.

[Mike]

JWZ's backup advice is good. Long story short: if you have X TB of data you care about, then you need 3X TB of storage to keep it safe. If that's too expensive, then don't expect to keep your data safe.

http://tinyurl.com/2ovl8b

[Jim Meeks]

There are many good comments and sound ideas presented above. I attended a conference a few years ago in London on digitization of cultural collections. As someone above said, there is no uniform consensus at this time that I am aware of, but there are certain steps you can take to safeguard your material. The various institutions we visited had a number of storage methods once their original had been digitized. Some were kept on servers, or hard drives and others were put onto archival DVD's. Some places did both and all had backups of some sort. One place had their collection on three sets of DVD's. One was used in-house for various tasks and if it got damaged, there was a second on hand as a spare and a new backup would be made. The third was kept off-site for safety. Some of the people using servers had off-site digital storage maintained by an outside company. All the institutions were aware they would be migrating their media into other forms and onto newer media as it came along. If you have analog materials, be sure to use inert and acid free materials and keep things dry and cool. No attics or basements please.

[Randolph]

I think this is a topic that deserves some serious research funding. Magnetic media are inherently non-archival, and all the popular optical media (CDs, DVDs, and so on) depend on dyes, just like color prints. There are better and worse media in this area, but there's nothing we have a good reason to believe is going to last for ages. Meantime, I keep three backup drives and swap among them--one is always off-site in a safety-deposit box.

[Bill Mitchell]

Edit savagely, delete with vigor, and if there's anything left worth saving (and unless you're as good as Ansel Adams there probably isn't), transfer to a professional archival data base and let them worry about it.
Another option: Print and sign everyting worth keeping, and then delete everything from your computer and retire (or die) happy.

[Shadzee]

Great subject to discuss which goes way beyong technology and photography ;-)

I suspect the vast majority of people who read this blog are NOT pros. There is a major difference between archiving for a corporation whose life is much longer than an individual.

Personally, I take thousands of photos, use Photoshop to experiment with many different techniques (PSD files are HUGE compared to DNG,RAW), hence create hundreds and hundreds Gigs of Data. I use external Hard Disks (quad interface USB, FW400, FW800, eSATA to guard against technology changes) to store and archive data. I always have double backups of my current year's work.

But as some one who does not make money using photography, I know after I die all of these will most likely be thrown away! Garage Sale anyone? The kids at least will eBay the lenses, but the data? Can you say landfill?

So the data backup is for while we are around and active... that's about it. For most of us there is no need for a 200 year data archival media ;-) No need to think about data format changes (jpg, dng,...) as long as we deal with it during our active photography period.

The only thing that maybe important for us is if we want the people we care about enjoy our passion for photography after we're gone. More and more that would manifest itself in PRINT and maybe if you're more technology aware... Digital slide shows and videos.

My personal solution: Making photo books. That is possibly the BEST archival media there is ;-)

[Mike J.]

I'm also leery of solutions that depend on companies or corporations to migrate the data. We've seen many times in the past that even large, prosperous companies eventually die, and usually their archives die with them. There is a photograph in one of Michael Carlebach's books of a couple of guys sitting in front of the archives of a New York newspaper of the 19th century—an entire wall of files full of negatives. The negatives were discarded when the newspaper folded. Another story I know firsthand is that when the Illinois Central Railroad relocated to smaller quarters, much of its archives were discarded, including more than 13 large boxes of annotated infrastructure photographs going back to the 1800s. Into the dumpster. At the very least, if these had been disseminated to railway enthusiasts, they might have survived.

Mike J.

[Tim Bradshaw]

One thing that will become increasingly important as data volumes increase is the error rate, specifically the undetected error rate of the storage media. Disks, for instance, do, on occasion, fail to report errors, but instead either return, or store junk data (or in the worst case write the data somewhere else than it was intended). This kind of silent failure is toxic for conventional RAID solutions, as they tend to rely on the disks not lying. The solution is storage systems which store checksums of the data, separately from the data itself. Most systems do not (yet) do this - Sun's ZFS is an exception. A DIY approach to the problem is to make your own checksums of each file you want to store.

[Ben Curtis]

Bill - "Edit savagely, delete with vigor", I have to say I totally disagree at least for professional use. "Delete nothing" is a better policy - fact is you do not know now what might be important in the future.

Tim - ZFS is amazing, and hopefully will become more widely adopted once Apple incorporate it into their available filesystems, as should happen with OS X 10.6 (server version at least). You can try ZFS right now on OS X, but it's quite beta-ish and in its current form I would NOT use it for archival as there's still plenty of bugs. But once ironed out it is ideally suited as a photo archival filesystem.

[Ctein]

Dear folks,

I'm going to take exception with an off-repeated statement that wet darkroom prints constitute a good archival storage medium.

In an ideal universe that is true, and we do have examples that are proof of concept.

In the real world, the evidence is very much to the contrary. The MINORITY of old prints are in "like-new condition. With the uncommon exceptions of Kodachrome slides and dye transfer, pigment, and platinum prints, a substantial fraction if not the majority of viewable photographs from only 50 years ago are NOT undeteriorated. Whether due to poor choice of initial materials, inadequate processing, or unsatisfactory storage conditions, the bulk of real-world images have not constituted a stable archive.

While there is some reason to hope that current digital inkjet prints will actually be better, we do not yet have proof of concept; they simply haven't been around long enough for us to be sure we know and understand all the failure modes. Accelerated and laboratory-condition tests do not tell you whether something will prove to be a good archival medium; they merely tell you whether something *might.* The track record for real-world photographs should have taught all photographers that by now.

I can make some specific recommendations. If you are among the very small minority of photographers who only do black-and-white work, your silver halide prints will likely be archival if you print only on fiber-based paper that you know is free from brighteners and incorporated developing agents, if you process the prints to archival standards (and you have run the chemical tests necessary to confirm that your processing is archival), if they are lightly toned with selenium and treated with sistan, and if they are stored in enclosures that you are certain will not contribute to deterioration.

If you are doing color printing, Fuji Crystal Archive paper is a reasonably secure medium, especially if you use twice-longer-than-normal wash times.

For both of these media, the failure modes are sufficiently well understood to give one confidence in the stability of the images on the 50 year scale. Still, refrigerated storage in water-tight envelopes would be preferred; it will markedly slow any deterioration. Freezing is unnecessary.

~ pax \ Ctein
[ please excuse any word salad. MacSpeech in training! ]
======================================
-- Ctein's online Gallery http://ctein.com
-- Digital restorations http://photo-repair.com
======================================

[Ctein]

Dear folks,

This is a complex issue and people have been intermixing several situations, which I think makes it more confusing. Let me try parsing the problem:

1) Timeframe: it makes a big difference whether you're talking about archiving during your active professional life and archiving after that. They are two different problems. The best solutions are not likely to be the same.

2) Location: you may also be using different strategies for your on-site and offsite archiving, and you should have both. (One of the benefits of digital archiving is that you can have more than one "original;" you might as well take advantage of it.)

3) Access: what are you doing to ensure that when you want to find a specific photograph in 20 years you will actually be able to find it? Conversely, if you pop a random archive disc into your computer 20 years from now, what are you doing to ensure that you will be able to QUICKLY figure out what the photograph is in each of the files? (Opening and viewing all the files does not meet the definition of "quickly.")

A library without any kind of a cataloging system (even if it's as simple as ganged proof sheets) is useless.

A final framing question is how much time and money per year are you willing and able to throw at the problem? (I'm not asking what you'dlike to throw at it, which is of course $0 and zero effort, but what you can reasonably expect yourself to do.) If you come up with a strategy that exceeds that, you will end up not doing it despite your best intentions.

~ pax \ Ctein
[ please excuse any word salad. MacSpeech in training! ]
======================================
-- Ctein's online Gallery http://ctein.com
-- Digital restorations http://photo-repair.com
======================================

[Ravi Bindra]

Call me retro - why not back the images onto film? We know it lasts for hundreds of years. And we know what to do with it to see the image again.

For the technology we need to do the reverse of a scanner. So basically get a film camera and aim at a high res image.

[Eric]

With digital files, it's important to note that there's two issues to keep in mind:

1. The physical storage - how much capacity it has, where it resides, and what the failure rate is.
2. The format - will you be able to read the storage device in 10, 50, 100 years? And will you be able to read the file format themselves in such a time? (For example, think about someone who might have an archive of Word 1.0 documents on 5 1/4" floppies, and what good it does.)

On the bright side, hard drives seem subject to some version of Moore's law; they're doubling in size every two years. As such, I don't think capacity is a huge problem. 1 TB drives are currently affordable; in another year or two, you'll be able to get 2 TB drives, then 4, then 8. (RAID or a Drobo isn't a bad idea to protect against hard drive failure).

I think if you simply keep *enough* capacity and make a habit of retiring your older/smaller drives and migrating the data to the bigger ones with some regularity, you should be fine. (And keep up with changes in hard drive technology - like the move from IDE to SATA to whatever comes next).

The other issue is keeping multiple copies, preferably at different physical locations - in case of catastrophic device failure. This is more difficult, but I think online storage will become an increasingly viable option - bandwidth will increase and Amazon S3 and competing services should get cheaper with time.

I've never been a fan of optical media; it was a good enough stopgap back in the day - but burning DVD's are time consuming, difficult to keep track of, and degrade with time. It's much more difficult to make sure a pile of DVD's is still good rather than a few hard drives.

File formats are trickier - I don't think any are "future proof". So just stay on top of changes in the industry, and if it seems like software that can read your camera RAW format is going the way of the dodo, it'll be time to convert your archive to something else. But Moore's law again makes this easier the longer you hold off on it, even as your archive grows.

Personally, I have a ~1 TB archive. I keep one Drobo on my desk and another at my parents' house, which I can VPN to, in addition to a couple of external hard drives. I keep two local copies of everything, as well as one remote one. I'm not too worried that Nikon's RAW format will go out of fashion in the next few years, but if it does, I'll just make a project of upgrading my files to something else.

[Ctein]

Dear Tim, et al.

I've got a simple and accurate (in so far as I know) way of verifying my archives and proving that my storage media are readable. I open up Photoshop, run Contact Sheet II, and tell it to make 8 image by 10 image proof sheets (scaled for 8.5" x 11" paper) of the archive disk (including subdirectories). Then I just leave it to run. If it's a DVD, it'll take less than an hour of unattended time. If it's a hard drive, I'll leave it overnight.

Photoshop is fairly fussy about files; it will usually report as unreadable files that many applications will open with just a localized pixel or line error. if Photoshop creates the contact sheets without halting, it's pretty certain none of the files are corrupted. To be even more certain I visually scan all the images on the contact sheets. If there are any serious errors, they will jump out.

Nothing is 100% certain; it's possible a bad file will slip past me and Photoshop. But my level of confidence is extremely high.

As a side benefit, I label and print out the contact sheets and put them in a binder, just like I did with my old film proof sheets. Not a really serious solution to the cataloging problem, but as good as I was doing with film.

~ pax \ Ctein
[ please excuse any word salad. MacSpeech in training! ]
======================================
-- Ctein's online Gallery http://ctein.com
-- Digital restorations http://photo-repair.com
======================================

[MESullivan]

I am an amatuer who still does a little photography for pay. As someone who had lost some very valuable digital photos years ago I became absolutely religious about backing everything up to CD/DVD.

When my computer died last year I purchased another and began to access my CD/DVDs. Lo and behold several gave error messages and were unreadable. I had tested each immediately after making them so something may have happened in storage. They were constantly in the dark in 68 degree low humidity air, stored with my negatives and slides. I have no idea what happened. None were over three years old.

I now am backed up to three hard drives one of which is always at another site. Everything is saved in TIFF and most in Jpeg also. At least what I consider worth saving. I have and will continue to edit ruthlessly just as I have with film.

Important family events, I make three print copies of the "good" pics, and keep one set for myself and distribute the rest among the family. That way surely a few pics of the family "way back in the day" will be preserved for future generations.

So as one who has had computer/hard drive/dvd/cd failures and spent several days of my life uploading CD/DVD files I recommend multiple copies on multiple hard drives at multiple sites and prints.

I think there is no easy answer to the problem. Everybody has to do what they feel is within their time/financial/ability limits.

[Mike J.]

"Conversely, if you pop a random archive disc into your computer 20 years from now, what are you doing to ensure that you will be able to QUICKLY figure out what the photograph is in each of the files? (Opening and viewing all the files does not meet the definition of 'quickly.') A library without any kind of a cataloging system (even if it's as simple as ganged proof sheets) is useless."

I can ruefully attest to the truth of this. I have roughly 3,000 books in my house (the number that were in the Library of Congress that was burned by the British in the War of 1812--replaced with Thomas Jefferson's personal library of 6,400 books, then the largest privately held library in North America), and I sometimes can't find what I'm looking for. I have insufficient shelving and no filing system other than memory, i.e., "I think that used to be over there." Furthermore, a good cataloging scheme would, at the moment, be defeated by the lack of sufficient, and sufficiently flexible, shelving.

This brings me to the 80% rule, first explained to me by the priest at a church I attended: he told me that when the nave of a church is regularly exceeding 80% occupancy, it means you're turning possible attendees away, because a nave needs a 20% vacancy rate to remain welcoming and flexible--above 80% full and some people would rather leave than sit down. Without much justification I've reflexively applied this basic rule to a number of other "container" situations--a print box should be 80% full, a bookcase should be 80% full, etc. I don't know how rigorous this is, but it does seem like some "extra" space is needed for a container to remain at optimum usefulness. I can say, at least, that a bookcase that is stuffed is not working at peak efficiency.

My books could well be thought of as a visible analog for some of the problems inherent in storing and organizing pictures....

Mike J.

[David A. Goldfarb]

When libraries acquire archives, usually a librarian, working with someone who knows something about the material when possible, starts by producing a rough index called a "finding guide," so that scholars can begin working with the archive before it is meticulously catalogued. In part, the work that researchers do with the archive while it is in this state may assist in the categorization of the materials, because people who understand the material will identify some items as more important and others as less important.

A good research librarian can come to know quite a lot about what is important in an archive, even if they don't understand what is in it in the way a scholar working on the material does. The best example that I know of is Vincent Giroud, former director of the Beinecke Rare Book and Manuscript Library at Yale, who put considerable effort into collecting the papers of modern Polish writers there, and seemed to know where everything was, even though it was all roughly catalogued, and as far as I know, he didn't know any Slavic language.

The point being that conservation and cataloguing are sciences unto themselves, and if you've got something you want to preserve and organize, there are professionals who know quite a lot about how to do it, and who might be able to give you some advice either directly or by pointing you toward some appropriate resources, and they are often quite accessible at public research libraries, museums, historical societies, and at local university libraries.

A few good places to look for information about preservation and cataloguing online are--

loc.gov --The Library of Congress, which has everything from technical articles for professional librarians to popular guides for scrapbookers.

palimpsest.stanford.edu --A major resource for librarians about conservation of materials.

www.digitalpreservation.gov --The Library of Congress's website about the preservation of digital materials and digital archiving of non-digital media.

[bobdales]

Digital rules:
Use whatever is current for your production use. Keep multiple copies in multiple physical locations. You must change to the next "solution"... that means new software, hardware and media. The key is to be able to keep up with the new technology and to migrate all your archives. This is not automatic. The good news is what works for one image works for thousands.
ALL HARD DISKS FAIL. ALL DVD/CD's FAIL. ALL FILE FORMATS DIE. ALL HARDWARE/SOFTWARE PLATFORMS (THANKFULLY) DIE.
If this is unacceptable try film processing and silver gelatin printing. Or, better still, change your name to Leonardo and have Ctein put your dye transfer image on a ceiling.

[Mike]

I suppose in a hundred or two years, it we're still here as a technological society or functioning species, close to no one will be interested in 99,99999% of the images being stored. If living is a process of "letting go" then perhaps we shouldn't be so anal retentive and let the past fade away.

[Andreas Weber]

"ALL HARD DISKS FAIL. ALL DVD/CD's FAIL. ALL FILE FORMATS DIE. ALL HARDWARE/SOFTWARE PLATFORMS (THANKFULLY) DIE."

Exactly. Redundancy and migration are the only options.
I've often seen people claim their DVD based backup was archival. ("Gold DVDs used in a selected high quality burner, verified after burn, never had a failure!") But I've yet to see one of them claim "I've just verified 1 TB of files on > 5 year old disks by comparison against checksums or known good copies! ..."
As several other commenters I keep my archive on three sets of external hard disks, one of them always offsite. The contents of the disk in use are occasionally verified against checksums. (One nice thing about Lightroom is that it never changes your original files but only the "processing recipe" - so the checksums always stay valid.)

[Graydon]

Couple-six notes, from someone who is concerned with this professionally. (I "own" a corporate document database that's used to generate shipping products; people would be upset if it got corrupted or stopped working...)

The way to model reliability--how you decide from a theoretical basis if a system is reliable or not--is still an area of active investigation and disagreement for digital storage. (Never mind archival digital storage!) So everything collapses to thumb rules:

Three copies, minimum. [1]

It's not a backup until it's somewhere else.[3]

If it isn't automated, you won't do it.

Disk is free.[4]

Nothing is archival, so copy everything forward.

Formats that conform to open public standards make it much more likely someone else will have already written the converter when you need to switch.

Keep the primary and the live backup on different filesystems.[2]

You can fit a lot of tape cartridges in a safety deposit box.

[1] "Copy" in this specialized sense means "different media, different file system[2], different location"; no matter how many instances of a file you have, it's not a copy if it's in the same case (and dependent on the same power supply hardware) as the other instances.

[2] Not 'filesystem' in the sense of 'defined mount point', but 'code that does the fundamental organization of the content on the hard drive'; in my case this is XFS and ext3. Macs have various options; Windows users are poorly served in this area. The point is to avoid having a bug in the filesystem code corrupt everything.

Industrial file systems like XFS or ZFS are worth serious investigation in this area; they are built by and for people who must not lose anything from enormous amounts of data.

[3] Backups most typically fail because no one tested recovery; sure, you think you made it, but can you get stuff back from there?

[4] Not literally, but compared to the cost of figuring out how to be clever with storage effectively, yes it is.

In terms of actual stable archives, well, we haven't got a good medium yet. Tape is probably the least bad of the available technologies.

[Mike J.]

Graydon,
That does it, I'm going back to film. [g]

Mike J.

[Graydon]

Hi Mike --

I seem to recall that film requires a lot of refrigeration, controlled humidity, persistent total darkness, and other tricky and expensive things to arrange to get it to really last. :)

My own set up -- for a hobby photographer with a paltry accumulated 315 G of images -- goes like this:

Actual hardware RAID card, and a nominal terabyte RAID-10 array in the computer. (Remember that RAID-5 is a hazard; disk failure times are not independent if you buy the disks together, and the odds of getting a RAID-5 array rebuilt after the first failure and *before* the second failure are remarkably bad. RAID-10 is mirrored, so there's always a complete copy on 2 drives. The hardware RAID card will cheerfully take a new hard drive and just deal with things if you've got a problem with one of the existing ones, as well.)

The RAID gets backed up, via eSATA (much faster, simpler, and generally better than USB), to a couple of external hard drives. This is automatic (cron job running rsync, in my case, but there are a *lot* of options for how to do this) every morning.

Long term storage is a mix of CDs (for a very few photos with personal significance that I'm being massively redundant about), prints (same case as CDs), and tape. VXA tape is expensive in terms of hardware, but for price/GB it's very hard to beat in terms of the ongoing media cost. And "expensive" in (home!) computer hardware terms is also "about what the Pentax DA* 300mm lens costs". This shouldn't be difficult at all from a running-a-business point of view.

Any decent backup plan has to look at risk, and what the thing being backed up is worth; for me, well, if the house burns down that will suck, but the photos I can't bear to lose are in that safety deposit box in a seriously redundant way. For a pro, if the office burns down and takes ten years of digital portfolio with it, that's presumably worth a great deal more effort to avoid.

Archival, well, right now, the only thing anyone _knows_ works is copy-forward. That's not likely to change, even if a novel technology (really cheap NAND flash, holographic, etc.) appears with an arbitrary data lifetime, because bus and interface standards are already much less long lived than the media. Powered down hard drives ought to last a good long time, somewhere between twenty and fifty years; same with stored tape cartridges, only there it's likely "at least fifty years". The problem is that the hardware standard for plugging the media into the computer rarely lasts 10 years before it's been replaced by something newer/better.

Of course, the US Navy did solve this, back when they were worried about preserving information in the face of nuclear war; punched mylar tape and a purely mechanical (hand-cranked!) reader. Be a bit expensive, though. :)

[Lawrence Siegel]

Just a note to add a little personal experience to the archive mess. Back in the day, digitally speaking, I used an image-editing program that saved my scanned film images in a proprietary format. The program no longer exists in an XP or Vista format, as far as I know, and I doubt I would buy it just to convert to .dng or another (I hope) non-proprietary format. I still have the original print film negatives and have them indexed by date and content, and I still have film and reflective scanners, so all is not lost.

I also started saving scanned files on floppies, but suffered a catastrophic loss of discs from a roof leak, then went to ZIP discs, and survived the failure of an early Zip drive fairly well, then saved the zip disc files to CD format. But early cd's didn't always read from one drive mechanism to another or even on the same drive - sort of ephemeralism, I guess.

Next I went to cd archives, first printing out a directory of the contents and storing it with the disc, then just writing the contents in summary form on the discs themselves. And then on dvd's. But as some of your commentators noted, reading the discs to determine their content is slooow.

I've only been doing the digital dance for about ten years, first with film scans and then with DSLR's, and all these changes have occurred. But I do have proof sheets and negatives, and slides going back to the early 1960's when I returned to photography and those have been my archives. Ektacolor and Kodachrome slides, however, have not faired well, fading, color-shifting, and bubbling in as little as thirty-five years.

Much as I believe posterity needs my images, I don't see a practical way to archive them, including all those mentioned in this topic. I doubt if my son or granddaughters would be able to find or decipher any of my current storage techniques, which now rest principally on four external and two internal hard drives.

The technology, governed by an application of Moore's law, is changing and must be expected to continue changing, relatively drastically and quite rapidly. The nature of the solution, particularly for individual practitioners of the art, does not appear to be on the horizon, if it even will ever exist.

But remember Brady's collection of glass plate negatives. How many of the total have survived the vicissitudes of time and human nature? As was said a war or three ago, hope for the best but keep your powder dry!

[cb]

Mike, do you forgot your own answer? Its here: http://www.luminous-landscape.com/columns/sm-10-06.shtml

In your OWN WORDS: 7TB data? Well Edit like a #$%!&er!!!

[Hudson]

The perfect archive needs to be man readable without special equipment, silver images meet this requirement.

A lot of the research I do, I run into the need to use microfilm, unfortunately microfilm usage is dieing, but luckily there are just 3 main types, 35mm, 16mm & microfiche.

But of course everyone had to build a better mousetrap, and there are multiple types of propriety cartridges for microfilm readers that are no longer made, but luckily most of the time you can open the propriety cartridge and rewind the microfilm on to a open reel. I have even gone so far as to read microfilm with a loupe and a flashlight.

The main thing in archiving is making sure it can be read in 20+ years without a machine that is no longer made.

The classic case we use to demonstrate this is the 1960 US Census, the data is all on magnetic tape, and IBM last made that tape drive in 1964.

[improbable]

Quite a salad of answers, delving into archiving in general I think... Ctein is correct that what timeframe (and purpose) is the critical question.

If you'd like all your work for as long as you're working, then copying it along with you is probably OK. Moore's law is your friend. When you stop caring it will go away, as all things do.

If you'd like your grandchildren to find some things, then lots of prints are a good idea, give them away to all your relatives. As is a website, archiving of these is improving... just keep it simple!

If you'd like everything you ever shot to be catalogued and available a century hence, then you should spend your time getting famous: HCB's will be, yours probably won't. (That or endow a foundation, but I'm not sure for how long you could hope for non-famous work to be preserved.) A lesser version of this would be to get a book or two published, they'll last a long time in big libraries.

[richard leonard]

As a professional photographer, I have grappled with this problem. I eventually came to the conclusion that the only secure way was multiple copies in multiple locations, periodically updated & secured from fire & theft... So, if you are a professional in New Zealand, you can buy into my solution. Sorry for the plug, but I really think this is the only way!

[Mark Tomlinson]

All this reminds we why I went back to shooting film. After baulking at having to re-archive my raw files yet again which already take up several boxes of DVDs I just gave up.

[thalarctos]

Some years ago, I worked in IT for a Fortune 100 corporation. Part of their approach to backing up their mainframe was to do it in triplicate: one copy in the next room (protection against accidental file deletion or disk failure), one copy in a fireproof, climate-controlled warehouse at the opposite end of the site (protection against fire or water damage to the IT building), and the third copy in a secure storage facility a few hundred miles away (you figure it out).

[Will Sadler]

I have a friend with pictures of her family from the last half of the 19th century in a basket on her living room table, each in its own beautiful little case. Mostly they are ambrotypes, but there is one little gem of a daguerreotype. It easily dates from 1860 and still looks fantastic. I'd suspect that a platinotype is probably the longest lasting medium of all. Died out in the 1920s I think.

[Riddle]

My family has about 7 years worth of digital photographs all stored in iPhoto and I am just venturing into the world of DSLR's, RAW and the 'real' digital darkroom. Having a couple of on and off site backups of my iPhoto library has been enough thus far, but RAW seems to be a completely different beast when thinking about archiving. jpeg is fairly ubiquitous, RAW is extremely volatile since it is camera manufacturer and model specific.

For family photography I have to agree with Charlie H. Photo albums (or in my case, bound photo books which are easily produced from online vendors) distributed to the whole family takes care of this whole quagmire. These photos will only have a relevant meaning for so long just as visiting a family member's grave site generally lasts only a generation or two. As long as they are shared and enjoyed through the most simple of mediums, the family photographer's job has been fulfilled.

Now back to figuring out how to archive my 15GB audio projects. The Smithsonian archival standard is still 1/2" tape since we actually know what happens to it after 50 years.