« Wednesday Open Mike: Eggsactly | Main | Correction: The Meaning of 'Reflex' »

Wednesday, 21 September 2016

Comments

There's another side to the ongoing manifestations of Moore's law that is especially relevant to still photographers. The cost of solid-state storage has continued to fall precipitously, such that a 2-Tb solid state external 'hard drive' is now just over $600. In my humble opinion this is now the ideal storage and back-up medium for digital images.

For years I have kept my photographs on external hard drives. These have increased in capacity and decreased in cost, but they remain fundamentally fragile devices with moving parts and a finite lifespan. I have had several fail catastrophically over the years, highlighting the need to have several copies. They also have relatively slow transfer speeds. As my photo library has gotten bigger I have had to migrate to more capacious drives, and backups have taken steadily longer to do. My photographs now occupy just under 1 Tb of disc space, meaning I really need a 2 Tb drive to store them all in one place and allow for ongoing storage. Even with a fast USB-3 cable, transferring the whole library takes close to 24 hours. And I like to keep three separate copies current, one off-site. Because I'm more than a little OCD. Yes, multi-disc RAID systems are automated and can take care of all this on their own, but they can cost serious money and require learning a new set of software tools. Life's too short.

I'm now using a pair of 2-Tb solid state drives, and they're fabulous. I can hide them in my palm, and transferring 25 years of images takes all of 90 minutes. Accessing images from these external drives is much faster than from my computer's on-board standard hard disc, so that's what I work from in Photoshop. Back-ups were never so easy, and the lack of moving parts promises much better long term reliability.

"..and nobody would really want to read what I would write about it."

Surely I can't be the only one who is, eh?

Another important technology at the "back end" of cameras is wireless transmission of files to phone and/or computer.

Dual card slots are all the fashion now. Leica SL and Fuji X Pro2 came out with them a year and half a year ago, rsp. Olympus has them in their new M-1 vII. I suspect there is some common chip that everyone is using, since the first three cameras with this in the mirrorless market all have UHS-II on chip #1 and UHS-I on chip #2. They differ a bit in implementation details. Leica lets you write to one after the other or to both simultaneously, and there is some mumble in the manual about keeping video on chip #1. The X-Pro2 and the Olympus M-1 Vii also let you put the raw files on chip #1 and the jpegs on #2, with the latest, Olympus version letting you assign "two different output types" to the two chips (with deeper menus, I presume, to choose them). The X-T2 is the first to support UHS-II on both chips for internal recording. The others can't just catch up with this with a firmware upgrade -- the UHS-II chips have more contacts and need a different connector, so Fuji may have gained a half step here. Watch for the point at which 4K video of full color depth and 10 bit dynamic range can be recorded internally, with the audio monitored by a headset plugged into the camera. This will get rid of quite a bit of external gear needed today by a serious video shooter. Watch Panasonic and Sony for the next steps in this race.

Hmmm. A more carefull reading of the less complete information about the Olympus OM-D E-M1 mk ii suggests that it may also support UHS-II in both card slots.

[Yes, it does. So does the X-T2. The X-Pro2 and the Leica SL only support it in the first of two slots. --Mike]

For video, think low-compression CODECS. Then think 4k video (or 4.5k, or 6k, or 8k).

Then run screaming.

Glancing through the estimator on the RED website, I can find examples of cameras that need to put 240MB/second someplace, and that's commercial products shipping today.

Most of them won't record to SD cards of any sort, which are both too small and too slow to be useful.

A terabyte is actually 1024GB.

I would say they have reached the "1TB" barrier, but have not broken it yet. Of course, "1TB" in this case has about 9.1% less capacity than a true 1TB.

A word of warning to those who believe SSDs (or flash drives) are great for backup:

They're not.

SSDs are less likely to fail catastrophically than mechanical hard drives, but they are more likely to lose data. They have higher UBER rates (that's "Uncorrectable Bit Error Ratio," not the rate at which they catch taxis).

SSDs need to be backed up, preferably to a good quality HDD or HDD array. Occasional archival backups to M-Disc are probably a good idea for stuff you really don't want to lose.

I have the Panasonic FZ1000 which can shoot 4K video and uses a UHS-II card. My very big brand name, expensive UHS-II card reader broke (I mean physically - it won't hold the card in the slot and is sealed so I can't fix it).

While in Bali last month, I browsed a dusty computer shop on the north coast and there on the wall was a strange NYK brand card reader marked as accepting UHS-II cards. Excited but dubious, I asked the price. How about the equivalent of A$6.50? I figured even if it didn't work, it'd be no great loss. But it works perfectly. I could even read Memory Stick, Compact Flash, MicroSD and MS Micro if I wanted to. Bargain! Sometimes you win.

While on the subject of video, although I shoot some 4K video on my travels, without a story it's just a random collection. You need to have a story line in your mind and shoot to fill the pages of the script. Four minutes of a Balinese dance is nice, but without the ability to use cutaways and closeups, it gets boring. So I've found trying to mix stills (which I'm not bad at) with shooting video as well, to be a wasteful distraction.

About 10 years ago the company I was at at the time was doing some consulting work for Sandisk, and they were talking about 1TB SD cards even back then. I don't remember the timeframe they estimated, but even so we thought it was crazy talk.

While CPU speeds have been going up, the changes are far less tangible and have been at least outstripped by processing needs in software/UI. The shrinkage in storage capacity is even more striking.

Compare Sandisk's new card to the first machine I worked on the design of at Sun Microsystems, in early 90's. It was a 30-drive RAID machine, with the then-astonishing capacity of 30GB (each drive being 1GB), all in the compact size package of about 24x24x8 inches and only weighing about 60lbs. As I recall, the only Sun customer at the time with 1TB needs for the *entire company* was FedEx. Fully loaded machines were so expensive that Sun couldn't even afford to set aside sufficient quantity to do a full scale test ahead of time with 1TB capacity.

Again Pentax gets no love with reference to dual cards. K-3 II released April 22nd 2015. 645Z with dual slots released in 2014.

Nice to see these late comers catching up ;)

SSDS are much more reliable now than they were just a few years ago. So much so that they are now becoming common in enterprise level storage systems (coupled with the falling price). Also their reliability is mostly related to how often the individual cells are written to, which in our case is not very often. I would put my money on an Ssd outlasting a hard drive by several years in our application. By the way,do not rely on simple Raid arrays to keep your data safe. there is always a single point of failure. I know, I used to repair them.

Way back when Canon supplied cards with their cameras. I got a 16*MB* card with my A400 - my first digital camera.

Recently, I had to use it in an emergency. I could get about 2-3 full size images on it! I had to reduce the images to post card size to get any quantity on the card.

Be wary of using SSDs and USB thumb drives for archival storage. They are built with flash memory, which stores data by charging a transistor for each binary bit (1 or 0). The transistor gradually leaks electrons and eventually loses enough charge to erase the data.

How long the discharge takes depends on the transistor's size and other factors. Older flash memory with relatively large transistors and single-level-cell (SLC) technology may hold a readable charge for 20 years or more. Newer, denser flash memory employs smaller transistors and multilevel-cell (MLC) or triple-level-cell (TLC) technology. Their data retention is worse than the older SLC flash memory. I wouldn't trust them for more than 10 years. To be absolutely safe, I would reformat them and recopy the data from another source every 5 years or so.

Hard drives are more prone to mechanical failure but store data magnetically, so they will likely retain the files for decades unless exposed to a strong magnetic field (which can also hasten the discharge of flash memory).

The enterprise-grade SSDs for computer servers that another commenter mentioned are certified for high read/write longevity, not necessarily for long data retention. In other words, the memory cells can be written and erased many times before the residual charge makes them useless, but they won't necessarily retain the data for a longer period. They are designed for high performance, not archival storage.

As I write this, the desk drawer beside me still contains the 2MB cards that I used with my first digital camera, a Minolta DiMage from 1997. This new 1TB card is equal to half a million of my old cards.

Tom is Right about not using SSDS for archiving. However in normal use they are still likely to last longer than a hard drive. Actually the data retention is again partly affected by the number of write cycles that have been performed. But there is no perfect media for archiving, especially as there is no guarantee that the hardware will exist to read your chosen media. It used to amuse me when there were advertisements for gold CDS that were claimed to last 100 years. Where will you find a cd reader in only 20 years time? Similarly there will be no usb3. There is only one solution and that is to continually re-copy onto the latest media. The digital world is ephemeral and it is worrying. There are some new technologies in development that will hopefully address this.

There will be no "usb3"? There is one today, and most of the flash drives I buy support it, as well as most of the external spinning drives. My latest motherboard supports a USB 3.1. Am I missing something?

The gold CDs remain a good idea, because if the media doesn't last there's no hope, but people have some luck resurrecting old hardware. Besides, modern BluRay drives read CDs and DVDs just fine, so I can buy new drives to read CDs 30 years or so on from when the format was introduced.

The comments to this entry are closed.