« Olympus a 'Hobby Business'? | Main | Curious About the M Disk »

Sunday, 10 May 2015

Comments

Archiving for the family history is, of course, a good idea, but I cringe whenever anybody, especially knowledgable photo experts refer to the photographs as MEMORIES. It seems to be an insult to the human mind. They are pictures, they are part of history, they are important, but they are not, and have never been, 'memories.'

[Aids to memory, then. Would you go that far? --Mike J.]

"By using a universal format like a TIFF"

Tiff may be a universal format , but not in a good way if you are thinking of a format that will be readable in the distant future.

http://en.wikipedia.org/wiki/Tagged_Image_File_Format

start reading at "Flexible options"

TIFF (and DNG) are universal formats for making a container that describes it's contents. In theory the contents can be just about anything a programmer pleases as long as they are described accurately
In practice it's not even that. There are a lot of unreadable by recent software tiff files out there. I have one program that I use all the time that generates tiff files that photoshop can not read, just because of (i'm guessing) a signed integer bug .

There may be a subset of tiff that would be good for archiving, but I don't know what it is.

Someone who does know is probably going to read this and they should jump in with more detail

Oddly enough, next week I am visiting friends who are parents for the first time. They have no camera as such, but do have smartphones. My gift to them are two very solid photo-albums with 'archival' pages, and a small open account at a laboratory in their city. I keep suggesting that they print the pictures they want to keep 'forever', so maybe this will help.

I chose to do this as a book (less than 1000 images!) called Generations and made copies for my children and grandchildren. Removed the angst of digital storage. I'll do a II if ...

I agree with Michael Bearman; prints have the best chance for long-term survival. One need only spend time gazing onto the faces of prints dating back to photography's beginning -- genuine Julia Margaret Camerons or Henry Fox-Talbots -- to see that truth. And with today's archival media who knows how may centuries the world might have to face our homely mugs on paper?!

But Michael Perini makes a good case for creating secondary archives digitally. But I, too, have become very wary of expecting the world to sustain gadget and physical digital medium compatibility. The notion of maintaining virtual off-site archives is now a practical reality with the leaps in consumer high-speed data communication and "cloud" storage services. I, myself, have just begun preparing to create a cloud archive of key images.

But I close by again saying that prints are your best hope for immortality.

A very simple additional backup for the few most treasured pics is to email the jpegs around. Many family memories for decades were small prints. Highly compressed jpegs that print well enough to this size would be much loved if all else has gone.

Two years ago my aunt emailed me a picture of my father I never knew existed. A simple scan of an old b/w print, him standing in his RN uniform, name of ship on hat, in Ceylon (as it was then) in WW2. Image Quality 1/10, Image value 10/10.

The only way to preserve your important pictures for the future is to print them on decent paper with decent inks. Paper is a technology that once it has been done, you need nothing else to access it other than your eyes and some daylight. Can't get any better than that.

What you actually want to do is make sure this archive is in, and gets updated to, formats that modern tools understand. (It's your family album so presumably you look at it every so often anyway. No as onerous as for old tax records.)

Then, have multiple copies stored at multiple sites. Family members, safe deposit boxes, and the cloud all make sense.

Those issues are much more important then whether the flash chip is in a fancy aluminum case or not.

There is no point in being the family archivist if the digital and/or hard copy that you've spent so much effort "saving" are locked away where it is difficult to share them with the family.

At least one copy needs to be on-line so all members and generations can own the memories.

Final thought: as important as the image is, the metadata pertaining to the image is just as important. I have literally hundreds of images from a hundred years ago and no-one living knows the 5 W's (who, what, when, where, or why). While this information is still known, it needs to be stored in the image.

Nearly everybody in the world has a device (smartphone) which can take and view digital photos in jpg format.

Fears about jpg files becoming obsolete like old tapes and punchcard data don't recognise that the world has changed.

Hi Mike

I'd like to point out that Solid State Drives will lose data without power and manufacturers will normally state a retention period of three months. There is good coverage of this here

https://blog.korelogic.com/blog/2015/03/24#ssds-evidence-storage-issues

This focuses on hard drive SSDs but will likely apply to USB SSD as well. So before entrusting anything to long term storage it's probably best to check the retention period!

I couldn't find anything on the Corsair you linked to but I did find this:

The Sandisk Memory Vault: http://www.sandisk.co.uk/go/preserve/

Hopefully there are others addressing this issue.

You should also consider a fire and waterproof external hard drive. ioSafe is one company that sells them. I can't personally vouch for them, as I've been fortunate enough not to put them to the test, but it is extra protection for not much extra money (compared to a normal backup drive).

I would suggest putting the 1000 family pictures on the 100GB
M-Disc that should be available by the end of May.
www.mdisc.com

We live in a remarkable moment. At long last, the truly paranoid will be rewarded. :-)

Once upon a time, the difference in archival experience between the very careful and the careless was small. The careless shoved pics in shoe boxes and cheap albums and gave it no further thought. The careful used special album papers, stored their pics and slides away from too much heat and humidity and carefully curated their content. But for the most part the experiences of loss for these extremes was nowhere as large as the difference in effort and cost, at least within a generation.

Now, in the digital age, the very careful (as Mike P is clearly a member) can, through regular review (say every 7 years to check on format and media), preserve such content for a full lifetime against countless loss events that the previous careful person could never hope to defeat w/o many expensive copies. Of course, even the careful still need a caretaker in the next generation that is equally paranoid - that is harder to find. Useful paranoia comes through experience.

Currently, the careless are doomed. Their photo sharing site will go out of business, their iPhone account will be hacked and their home PC will lose drive content or fail to backup. Or a silent data corruption will have them happily making copies for years of data which is no longer accessible. They've never been more vulnerable to losing their entire life record. They "know" they should setup a good backup, but they will never get around to doing so.

The expected outcomes for the two groups over a lifespan is now enormously different, I believe.

For me, anything digital that I truly care about has at least four copies on different media and mechanisms and some copies are in different geographical locations. Furthermore, files are check-summed so that if data corruption occurs, I can detect it before I blissfully make bad copies. With luck, only one copy is bad and I can "repair the DNA" from a good copy. I've already made one technology transition from CD-ROM to flash memory.

A circumstance in which I "lose it all" is probably one neither myself nor my family are likely to survive. Or, if they do, finding old family pics and tax returns won't be a priority. (Think: Yellowstone eruption as an example).

And yes, that means like Mike P did, you must do the hard work of saving those 10, 100 or 1000 most vital images (as well as what other critical digital documents?) - curate the critical. And you must do it year after year, decade after decade.

An attempt to secure *everything* is going to be very expensive and very difficult and it won't happen consistently over the 40+ years you are caretaker.

Severian

PS: Why yes, I do work in the computer storage industry! However paranoid you are about data loss, I'm willing to gamble, you are not yet taking enough precautions. Mike P should consider having at least two copies of his archive, and in blunter terms - make sure the same tornado storm system is unlikely to get both copies on the same day.

Solid State drives aren't a good choice for long term data backup. They are far more likely to corrupt data in a relatively short time. See this thread on Slashdot --

http://rss.slashdot.org/~r/Slashdot/slashdot/~3/w3TqMAq41Ws/enterprise-ssds-powered-off-potentially-lose-data-in-a-week

I like this. I recently bought a backup drive and reconfigured my computer's hard drives to organize things better. I've also toyed with the idea of using my smugmug account as an additional backup, figuring that in the event of a disaster, I'll take jpegs, even downsampled to something good enough for 8x10s, over nothing. But I think it's also important to consider who's going to bother looking at my photos should anything happen to me, and so I like the idea of flagging a subset of photos that get backed up separately or additionally. My Lightroom catalog is nothing that someone who isn't into photography wants to muddle through. I'm not sure where I'll go with this idea, exactly, but I expect to incorporate it somehow.

Digital archives definitely require ongoing maintenance. Both media and file formats will require conversion at one point or another.

On the other hand, it's pretty easy to take care of text and images well enough for a personal-size collection (under 10T, say) well enough that the digital archive is much better protected than you could keep a physical archive. I've been doing it over 20 years already (and have only had one file format become problematic on the image side: Kodak Photo CD).

As a result of which, if my house burns, or is flooded, or carried off to Oz by a tornado, my digital photos and my ebooks and my digital music files will survive, but my paper books and LPs and CDs and prints and negatives won't.

Looking at much longer periods, there aren't many cultural artifacts over 2000 years old that haven't had several centuries of "benign neglect" or worse somewhere in their history. How that will work out for digital data remains to be seen, but is a legitimate place to worry some. Archive.org and such organizations are making one kind of attempt to help.

I don't know about a better way, but another, much more permanent way to save those essential family photos is to self-publish them in a book. The advent of Blurb.com and other similar print-on-demand services has made high quality book production available to everyone.

Advantages are:

• The great longevity of ink-on-paper printing (assuming quality paper);

• The ease and pleasure of holding and turning the pages of an actual book over looking at a screen;

• The ability to produce multiple copies whenever needed with print-on-demand;

• Lack of concern with obsolete viewing software and hardware.

I know these books aren't inexpensive, but what family wouldn't be enriched by a book of its own family history in photographs?

"Segregating the important pictures from all the rest is just as important a part of a Thousand Picture Project as redundancy and protection, and will do just as much—or more!—to preserve your family's history."

Ah, I see. Well, how about a folder on Google Drive then, full of TIFFs and with read access granted to relatives? Not as a complete solution, but as part of one.

50 years in the future, relatives who have not been born yet are going to need context for the photos. It would be a good idea to include some text files with the stories that go along with the photos.

Sorry for the multiple submissions.

There are far better options for cold storage than a flash drive. USB flash drives are suitable for data transfer, but not for long term archiving. I've seen too many flash drives fail to recommend them as a viable option. Optical disks are a safer bet. Check out the M Disk.

The comment by Bruce Mc is not off point. You can get a crash plan account and only use it to back up specific folders. Data centers take redundancy and backing up very seriously. Important data belongs in a data center where the pros can look after it.

Facebook is probably the safest/cheapest long term storage space for your 1,000 most important family photos.

I think the main problem here is selecting the 1000 images! I just couldn't do it - and part of me thinks I shouldn't do it as I would, in effect, be editorializing my family's life.

I do have a backup process on to hard drives duplicated at home and off site, but I have recently been taking advantage of Amazon's Prime free photo storage to make a copy of everything (4tb and counting).

The advantage is my kids (and ultimately that's what preservation is about) can access it easily, wherever they are - unlike many systems it lets you preview images so they don't need to download everything, and in now time it allows you to share specific images.

OK uploading 4tb takes a considerable time, but I just let folders upload as a background task.

I said "free", but of course you do have to pay for a Prime account ... but I think many people would cover that with the savings it offers in free shipping.

Of course, Amazon could change the nature or cost of the service tomorrow, so it isn't a perfect answer, so I'm also continuing with my physical disk backups - but I think these will be more difficult for my family to access.

It wouldn't hurt to burn your 1000 images onto a few archival DVDs, such as those made by Taiyo Yuden (JVC). These discs could last 100+ years.

I do this. I call it "syncing the archive of selects to every iOS device that we own."

I do it with JPEGs and not TIFF, but at high quality the practical results are about the same. Every device is then backed up separately to servers that hold such things. This is, of course, completely separate from the various on and off site backups that I make. But it's the most immediately accessible form of the "important" pictures in the archives.

And, as a bonus, I get a nice set of screen saver images too (on the machine that does the syncing).

Yes, I realize that iOS devices are not archival. But at some level they are only a bit more than a battery powered flash drive (128GB!) with a built-in picture viewer.

This is an interesting idea. My own version involves a "greatest hits" book of photographs of my children and their cousins, which gets distributed within the family every year at the holidays. I would say that there are somewhat fewer than 1,000 images in these books by now. I started to write that I would grab the books on the way out of the house if it were on fire, but then I added mentally, "well after my family and the cat . . ." and then realized that having any priority listing something else would probably mean that the books would get left. Time to order another set of copies for off-site storage. ;-)

I thought this was a settled issue. Gold-foil rewriteable archival DVD's are about a dollar a piece, and are anticipated to have a 200-year lifespan.

Two hundred years from now is a lot longer than most anyone alive today on the planet is likely to be remembered. For the 99.9% of us who are not recognized photographers, and really can't imagine ourselves to be the next Vivian Maier, that should be plenty good enough, shouldn't it?

I mean.... if the reply to that question is no - the solution is not a better storage medium, it is some sessions with a shrink.

The problem with all the archiving solutions mentioned in the comments is that they require ongoing effort to keep current. Most people I know (including me) are really bad at routine back-up operations and are even worse at remembering to take archives off-site to protect them from fires.

My suggestion is to go with cloud storage. If you use Google or Microsoft, you can have your computer archives automatically backed up whenever you make a change. These are reputable companies who take far more care with our files than we will ever do.

Microsoft has a deal where subscribers of Office365 (less than $100 a year) get unlimited off-site storage. I currently have over 2TB of photographs stored on my Microsoft OneDrive. The added benefit of using OneDrive is that Microsoft keeps all versions of every file, so you can recover if, for example, you get hacked by one of these companies that encrypts your files and holds you to ransom.

Google also had low-cost archiving (e.g. $2/month for 100GB). I also use Google storage to store some files.

Another benefit to these cloud storage solutions is the ability to share files from one computer to another and manage the syncing seamlessly.

Cloud storage is cheap, seamless and works with very little set-up and know-how. The beautiful thing is that it doesn't require any effort once it's set-up. It just keeps working.

The long term trouble with any type of digital storage is that even if the data survives intact no one will be able to read it. Do you really think that in 20 years time there will be a DVD drive available to read that archival disk? Definitely not. Remember floppy drives? Will that cloud storage still be around? Doubtful. Is it possible that TIFF and JPEG file formats will still be in use? Possible but not certain. Only prints made with archival materials will survive.

An important point about cloud archive/backup services: any one such service only counts as a single copy of your data, at best.(**) No matter how good the company's internal processes, there are always Single Points of Failure (SPOFs, in the lingo). At the very highest level, there's the possibility that the company or division will unexpectedly go belly up. Just as with physical media, it's good to evaluate the reliability and track record of a provider, but these providers are composed of potentially fallible parts and processes, just like the spinning hard drives of yore. ;-)

Following best practices, you'll want to keep a minimum of three copies of critical data. For my own backups and archives, this is usually the "live" copy attached to a primary computer plus two (or more) backups. Ideally, one of those backups (cloud or otherwise) lives off-site. This helps reduce the risk of loss due to catastrophic events such as fire, flood, etc. Businesses with critical data will often contract with data security companies, who essentially keep climate-controlled safety deposit boxes with a service that periodically comes around to swap lockboxes of old backups with lockboxes of new backups.

And in case anyone is wondering, it's definitely happened in the past that having all three copies saved my derrière from critical data loss.

(**) "At best": consider two different cloud backup providers that both use a common storage provider (e.g. Amazon S3) behind the scenes. If a major S3 outage or data loss event occurs, that could impact both apparently-distinct cloud backup services. If using more than one such provider, it's important to know whether they're sharing major back-end infrastructure.

I have to ask: what do people think they are going to use to read these optical disks in the future? Optical drives look like they are maybe 5 years away (at most!) from going the way of the floppy disk drive. I would think that optical disks are one of the RISKIEST options right now, no matter how long the media are theoretically readable.

Regards,
Adam

Does the M stand for Magical? Go to the website and tell me your bs meter didn't go off. The poster GingerBaker is on to something.

Dear folks,

This will be the first of several posts on this subject, because I want to keep subtopics compartmentalized, and I'm still collecting data, and, well,... longissimus, non legi !

Regarding prints vs. digital storage, many, many posters are conflating threat and risk, scope of loss vs. likelihood of loss. They are different issues, and failure to analyze them properly is the main reason people make bad “security” decisions.

When people talk about how long (some) photographic prints have lasted, they're talking about the equivalent of threat. They're talking about the potential. This is a most common error in analysis––treating the extreme or full-blown case as if it were the norm.

To give a dystopian example: a dinosaur-extincting-sized asteroid hitting the earth will really ruin your day. It may kill off the entire human race. It will definitely kill off 99.9% of it, and it will likely drive society back to preindustrial times. And, yeah, if you're in the lucky 0.1%, it's an argument for having paper prints instead of digital media… but that's probably going to be the least of your worries. So the threat here is monstrous, bigger than any other that we know of… and we know it will happen eventually. It's not a hypothetical.

If you're only doing threat evaluation, this tells you that you should be plowing huge amounts of resources into figuring out how to prevent this, more than you plow into any other problem of human survival. Because there is nothing worse that could happen.

That doesn't take into account risk, which includes the likelihood of an event happening. In the case of photographic prints, we know from historical data that almost none, maybe one in 1000, actually do last for a century, regardless of what their potential lifespan is. It's like the number of humans who lived to be 110 years old. We know that can happen. You shouldn't be making your plans on the assumption that you'll be one of them.

Back to the asteroid. One of those muthas hits every hundred million years, give or take. Which makes the annual probability so low that pretty much any other imaginable human accident or disaster has a higher probability. Conclusion? We should put absolutely no money or effort into investigating this.

Well, what about threat times probability? That's one way to evaluate risk. But it doesn't necessarily give you a correct answer. Our asteroid-in-wait will kill an average of 1000 people a year. Worldwide, that's damn small. Most orphan diseases rank a lot higher than that. Deaths by lightning strikes are 25 times more common. So we should be plowing a lot more money into a robust lightning defense or curing all orphan diseases than a robust asteroid defense, right? (Note–– we don't actually know how to do either yet.)

Maybe. Maybe not.

Are you seeing the problems with simplistic analyses? Homing in on one particular failure mode or one particular aspect of the problem doesn't lead you to a correct answer. It merely tells you what you are most afraid of.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com
-- Digital Restorations http://photo-repair.com
======================================

Dear Scott,

That is a misunderstanding of the information on those two websites, which is, in itself, faulty. GIGO plus.

First, the KoreLogic page is wrong in several major respects. There is no difference in retention times between client and enterprise devices (you incorrectly quoted the enterprise device retention time instead of the client time BTW). The three month vs. one year difference is entirely due to the fact that enterprise devices are specced at 10°C higher storage temperatures, which, according to the models, produces a fourfold difference in life. Furthermore, taking that down to what is canonically thought of as room temperature in most people's homes –– 20° C –– one comes up with a specification life of four years, not three months.

But that's still wrong. That's the design standard that the industry is setting for specifying these devices. In other words, it is the minimum they are supposed to reach to be considered acceptable. In fact, they may do much better. Or not. But you are confusing a minimum specification with an actual operational result.

The author at KoreLogic has also made a fundamental error–– he presumed that the anomalous loss of data was due to an inherent instability in the device. Far more likely, it's simply that the drive failed. If you look at any website for any storage product, you can find complaints from buyers whose devices, regardless of their nature, failed the next day, within two weeks, within two months, etc. It doesn't mean all devices fail within that timeframe, not even most of them. In fact, very few do.

This matters in a worst-case scenario, which is what the KoreLogic author is concerned with––namely preservation of forensic legal evidence. He's got a real problem! Establishing the long-term storage of digital evidence in a forensically-incorruptible chain is a non-trivial security problem. Mechanisms exist, but ensuring that they, in fact, have not been corrupted, takes a lot of work.

In our case, that worst-case scenario is solved simply by having multiple duplicate archives. The odds of any one failing are low; the odds of all of them failing are infinitesimal.

The SanDisk webpage is only marginally better. It's entirely possible that their engineers know how to do proper accelerated testing, but that data isn't presented there. There are two big red flags. The first is that it appears they are testing at only one accelerated temperature and the second is that they have obtained a zero level of deterioration.

The problem with the first is that the Arrhenius equations involve constants that you either guess at or determine empirically. For complex real-world systems, the guesses are always wrong. Flash memory qualifies as a complex system; so, for that matter, do photographic emulsions. To determine the constants empirically, you need data at two different temperatures and then you can calculate the constant. But –– second problem –– it's impossible to calculate if the change is zero. According to the Arrhenius equation, that says that the system is stable indefinitely long, like forever.

You can finesse around that to some degree, but it's still a pretty crappy data. And you still have no idea if you have the constant correct. Actually, you can be pretty sure you don't.

Furthermore, you need at least three temperature data points to determine if the Arrhenius equations even apply to your system. They don't, in many real-world cases.

This doesn't just arise as a theoretical problem. For example, a great many sensitized photographic emulsions age much faster in cold storage (-15 to -20 Celsius) then they should according to the Arrhenius equations, and we think we understand the chemistry of those pretty well. Yet we see changes occurring at 10 to 100 times higher rates than you'd predict: grossly observable changes in the behavior of the films in 5-10 years (which should correspond to a week to a month at room temperature, where you don't see such changes).

I don't know what the answer is to flash storage lifespan. But I know that neither of these websites gets you there. Garbage.


pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com
-- Digital Restorations http://photo-repair.com
======================================

Why not just upload tiff or jpeg files of your best photos on a cloud service? Nowadays they a quite cheap and you may review or download your lovely pictures in any device connected to internet.

THANKS for the great idea! I don't think my wife would be able to find the pictures on the computer, or two backup USB external drives! Let alone want to look through thousands and thousands of images. I'll need to start handling my favorites in a new way. Thanks.

The comments to this entry are closed.