Coming Revolutions in Storage and Memory

We were talking about SD cards yesterday, and whether 95 MB/s (megabytes per second) is enough download speed or whether you need to pay the premium for 300 MB/s cards. Yes, of course, some people do need the latter—pros, videographers, heavy shooters, anyone on a deadline, etc.

As far as storage and memory are concerned, though, you need to know about two things, both of which were mentioned in the Comments. The first is called CFexpress, a possible new standard in portable removable media that's a big leap from what we've had up till now. The other is some even more exciting news in archive storage, an issue which has been a real headache in digital (and for certain analog media, such as the originals of movies) for decades now. It's called Project Silica, and it's just in the proof-of-concept stage now. But the concept is fascinating indeed. As are the concepts behind the concept.

CFexpress
Think your measly 300 MB/s SD card is the sh*ts? Well, yes, it is, but maybe not for long. CFexpress cards use Sony's XQD form factor and will be back-compatible with XQD devices, and can potentially deliver four lanes of 1GB throughput per lane. Upcoming examples of the Type B version of the 38.5×29.8×3.8mm cards can hold half a terabyte of data with read speeds up to 1,750 MB/s and write speeds of 1,000 MB/s, and the potential is for even more than that.

This is better tech. But as Kishore Sawh writes on SLRGear, "we’ve seen 'better' tech come and go in various fields before simply because they weren’t broadly adopted, and this was a concern for photographers and manufacturers. Make no mistake, adoption rates and market share can make or break a 'superior product.'" Truer words were never spoken. But that, in case you were wondering, is why the Nikon Z6 and Z7 and other Nikons use XQD cards—Nikon has already announced upcoming firmware updates to enable the Z6, Z7, D5, D850 and D500 cameras to use CFexpress Type B. Nikon is only waiting for the cards to become available before releasing the firmware that will enable them. (I don't know how many other current, or imminent, pro cameras use or will use XQD, but I'm guessing there are and will be more.)

Photo courtesy Lexar

A Lexar CFexpress card fits in the XQD slot of the Nikon Z7

More generally, it remains to be seen if CFexpress will be the storage media of the future, because it depends on adoption and acceptance. For now, though, I can't see how companies making cameras for professionals could fail to keep up with each other and transition to CFexpress—it's just so much mo' bettah. In the more distant future, it's certainly possible that cameras will involve a paradigm shift away from traditional handheld interchangable-lens standalone cameras—already, many amateur cameras are index-card-sized, quarter-inch-thick wafers (that is, smartphones) that can send their pictures wirelessly to the cloud for storage. Who knows if professional digital imaging will go that way too, or some other way? But assuming there remains a need for portable removable storage and a need for improvement in that area, CFexpress may be on the cusp of creating a decisive leap in the existing state of the art.

Microsoft Azure's Project Silica

Photo courtesy Microsoft

And speaking of the cloud, check this out. What you're looking at here are, on the left, the four reels of the 1978 movie Superman on celluloid, and, on the right, the same movie encoded on a palm-sized piece of hard silica glass. It's a technology developed by a sprawling Microsoft division known as Project Red Dog (2008) > Windows Azure (2010) > Microsoft Azure (2014). It's called "Project Silica." The successful transfer in the picture above is the proof of concept.

It's astonishing. Microsoft says "the hard silica glass can withstand being boiled in hot water, baked in an oven, microwaved, flooded, scoured, demagnetized and other environmental threats." Awesome/brilliant as all that is, it's not even the new technology's neatest trick. An absolutely huge problem in data storage, and one that looms even larger for the future, is that vast amounts of labor—and hence, expense—are needed to keep ahead of entropy, a.k.a. decay. In this case, for instance, "Warner Bros. proactively migrates content every three years to stay ahead of degradation issues." Just imagine having to do all that work just to keep Superman in shape! Then multiply it over all the movies Warner Bros. has ever made, then to all the movies anyone else has ever made, then to all the other kinds of information humanity thinks is as deathless and eternal and essential to culture as Superman....

Anyway, what's sorely needed is storage media that is "write once, read always" (WORA), so when you put something away you don't have to tend to it. Ant Rowstron, partner deputy lab director of Microsoft Research Cambridge in the United Kingdom, says, "one big thing we wanted to eliminate is this expensive cycle of moving and rewriting data to the next generation. We really want something you can put on the shelf for 50 or 100 or 1,000 years and forget about until you need it."

'Zackly. Here's a picture of my former WORA technology:

I haven't touched it in better than 20 years and all the image information is right there  awaiting potential excavation. This method isn't practical any more—once it reaches this stage it's stable, but it takes too much work and cost to get to this stage.

The piece of glass in the picture above contains 75.6 gigabytes of data plus error redundancy codes. How much work will it take to get that information to that stage? For the piece of glass pictured, probably a whole lot. I presume the eventual answer to that question is that the answer will evolve. Project Silica isn't currently intended for home storage or computing (or for your Lightroom catalog)—it's for the cloud. But if it turns out to be practical—and see those comments above about adoption and acceptance—it could eliminate the eternal headache of "write once, then back up over and over and over again at regular intervals until you run out of money or until some idiot drops the ball, at which point everything goes to hell pretty damned quickly," i.e., the current state of digital archiving.

If you want to know how it's done, you'll have to read Microsoft's article, released just a few days ago, because it's over my head. (I'm guessing you already know whether you want to read it!)

Mike
(Big thanks to Speed)

Original contents copyright 2019 by Michael C. Johnston and/or the bylined author. All Rights Reserved. Links in this post may be to our affiliates; sales through affiliate links may benefit this site.

Please help support The Online Photographer through Patreon

(To see all the comments, click on the "Comments" link below.)
Featured Comments from:

No featured comments yet—please check back soon!

Comments

[Sal Santamaura]

...Microsoft says "the hard silica glass can withstand being boiled in hot water, baked in an oven, microwaved, flooded, scoured, demagnetized and other environmental threats."...

But will it shatter if dropped? :-)

[John Camp]

A lot of vanity in all this data-recording stuff.

[Look on my Works, ye Mighty, and despair! --Mike]

[Dave Hodson]

Just read the article on Project Silica - very interesting. It's a bit ironic that digital movies are transferred to black and white film for long term storage and conversion back to colour. More ironic that the recovered movie is apparently not ideal since film can't record the 1's and 0's of the original recording.

[Richard John Tugwell]

I still use pretty old slow cards with c.24mp sensors - for example in my Fuji XPRO 2 I have a 16gb SDHC Class 10 which is so old that I had trouble reading these specs (I think that means 10 mb/s)

I don't shoot huge amounts of images and I always 'download'images via a card slot.

Never eally had an issue with speed, although if I needed to quickly show some wedding guests 500 images, maybe I'd be looking at it more closely

[Duke Groover]

They didn't mention hitting it with a hammer :)

[SteveW]

My WORA is Google Photos.

[Andrew J.]

Fascinating. If CFexpress can hold half a terabyte of data, why make it a removable media? I can see future cameras with soldered memory that would simplify the construction of the camera (no flimsy doors, bad connections and so-so weather resistance). As an added advantage, no more losing those cards. You could still lose the camera but that seems less likely.

[I always thought the same, but it gets pointed out to me that memory fails, and if it's replaceable then you just replace it. If it's built in then at minimum it has to go in for professional repair. --Mike]

[Kirk Tuck]

Mike, both of my Lumix S1 cameras have two slots, one for SD (UHSII) and the other, currently, for XQD. Panasonic have announced that a pending firmware upgrade will open those XQD cards to CFExpress cards as well. Already physically implemented and ready to go. But in the meantime you can use your choice of XQD or SD or both.

[Hudson]

Having seen storage formats come and go, I much prefer storage media that is man readable.

The 1960 US Census is on mag tape, IBM last made the drives in 1965.

To digitize/convert the Apollo files they needed to come up with a replacement for the whale oil used lubricate the drives.

I am scanning my images and storing them on NAS/RAID drives, but I will never part with the film originals.

Has anybody seen my paper tape drive?

[JH]

As someone who lives in the world of data transmitted on fiber optics, much at 100+ gigabits/sec and a single fiber can carry up to 128 individual signals at 100G - I keep track of many new developments. The MS Azure project is not the first like this. Almost 20 years ago, researchers stored a terabyte of data in a tiny plastic cube using holography. Unfortunately, it was never commercially feasible. But the storage of data on CDs and magnetic disks has steamed ahead - the original hard drive I had in a PC (I'll spare you details of the days when computers had tubes and 8K was a lot of storage) was 20MB and the size of a book. I have a decade old iPod with 60G magnetic disk drive that is the size of a deck of cards. My backup drives are a 3TB disk and a 1TB SSD - basically like a memory card.

But there are two things to consider about long-term storage. For magnetic or solid state memory, it can wear out. Have you heard about the memory in Tesla cars wearing out? https://futurism.com/the-byte/flash-memory-old-tesla

Secondly, there is the issue of there being equipment to read the data. A few years ago a project to recover old NASA data depended on finding several old racks of equipment that could read it (https://www.nasa.gov/topics/moonmars/features/LOIRP/) but the project died when the people working on it retired and no further funding was provided.

The MS Azure glass may endure, but will anyone be able to read it?

I have over 30 years of files and data backups but the older stuff is junk because there is no hardware or software to read it.

There have been hundreds of media used for data storage - see https://obsoletemedia.org/data/.

Want to archive a photo? Print it!

[Dillan K]

Project Silica looks like a God-send. It's too late to save so much of our musical history . . . .

https://www.nytimes.com/2019/06/11/magazine/universal-fire-master-recordings.html

It breaks my heart.

[Ken]

It'll be interesting to see if XQD/CFE/Whatever takes hold, though I have some doubts:

The number of new, non-phone cameras sold is declining each year and sales have yet to hit bottom. And how many of those new cameras selling today (or will sell in the coming years) use the new storage devices vs. the widely available (and cheap) SD cards? Maybe someone can provide numbers, but I would bet that a majority of new cameras still use SD cards. Plus, most older model cameras still in use are incompatible with the new technology. So unless the manufacturers need to sell only a small quantity each year for the new cards to be profitable, I don't see how they can last.

As for the glass storage device, all that technology won't help if, as others already noted, the glass holding your priceless movies will shatter when someone holding it trips and drops it!

[Robert Harshman]

Funny stuff, WORA. Back when I was a systems consultant I worked on a project for the IRS, a debtor collection system where you basically had to ignore at least 3 mailing before being added to the outbound call system. And for pretty much all of you know the government agencies that might contact you will NEVER call first. If you get a call it's about 110% a scam. Do not respond.

Anyway, the system we built base on bid requirements had a write only tape system. There was nothing in the requirements to ever read the tape. So we had a set of files that were being written and stored to never see the light of day again. It was a interesting time in just how much money our government seems to waste on a consistent basis.

[Alexv]

The Project Silica wafers look a bit like memory cards used in the 1960s Star Trek series. Every now and then Spock would slip one into his science console.

[Maris Rusis]

It is not enough that data can be stored in an eternally stable form. The data reader, its software, and its power supply needs to be eternally stable and available as well. Who today maintains a reader for old style floppy discs or computer punch cards?
Mike,perhaps nothing is truly archived unless it is directly human eye readable like your negative and contact print files.

[ian]

Compare the HAL 2001 photo with the Microsoft photo. Mr. Clarke seems to have been a bit clairvoyant…

H.A.L. 9000 - H-A-L is derived from Heuristic ALgorithmic computer”. Clarke more directly addressed this issue in his book The Lost Worlds of 2001.

"I'm sorry Dave, I'm afraid I can't do that."

">https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcTGapeGFS2djZ08BhgjyKnhkedZsYmt6OO1Y2enMdP6y-zuZoko">

[Mike O'Donoghue]

It's worrisome that we can't access data on 5" floppies anymore or the smaller 3.5" ones, either. CDs and DVDs are not much longer for this world – who still uses them for music or film storage? Formats change so rapidly these days. I suppose the digital era is still in its infancy. How many more formats will come and go? Will there ever be an ultimate format?

Will we be able to access any of these formats in the time span they're crowing about? Will we still have electricity? I'm thinking of the 10000 year clock project here. (https://www.10000yearclock.net/index.html) Will people (or whatever beings may be around so far in the future) really be interested in us?

[Tim Bradshaw]

If your WORA is anything involving Google or some other company providing storage over the internet, you should probably look up MySpace. It may seem like Google will be here forever: they won't. And even before they go away, they may just decide that whatever service you are relying on is no longer important to them and turn it off, particularly if you are mot paying for it.

More generally if your WORA is a service rather than a collection of physical objects, it isn't a WORA. Especially if it involves Googlebook.

[Rodolfo Canet Castelló]

CFExpress card are already available:

https://www.amazon.es/Tarjeta-SanDisk-Extreme-CFexpress-1700-MB/dp/B07WVDV3FG/ref=sr_1_15?__mk_es_ES=%C3%85M%C3%85%C5%BD%C3%95%C3%91&crid=3MCFJDJ4CA1O8&keywords=tarjeta+xqd&qid=1573126313&sprefix=%2Caps%2C324&sr=8-15

What are you waiting for, Nikon?

[Robert Roaldi]

We're slowly inching our way from the era of "big data" to an era of "way too much frigging data". We need curatorial AI.

[Scott Kirkpatrick]

I bought an XQD card for my Lumix S1 recently, and it's fast. It's also expensive, because Sony controls all aspects. So I was interested to notice that the S1H video model uses only UHS II slots and SD cards, which can also be fast, but are less expensive. And the same for the Leica SL2, which was just announced. The CFExpress containers replace small hard disks for mass video capture, which can run at several gigabytes per minute and over 29 minutes of shooting in video cameras (still cameras, for obscure reasons, must stop at 29 minutes). So personally, I can wait a bit for XQD.

[Dave]

Mike J. wrote, ". . . my former WORA technology . . . I haven't touched it in better than 20 years and all the image information is right there awaiting potential excavation. This method isn't practical any more — once it reaches this stage it's stable, but it takes too much work and cost to get to this stage."

AND

"Project Silica isn't currently intended for home storage . . . But if it turns out to be practical . . . it could eliminate the eternal headache of "write once, then back up over and over . . . at regular intervals until you run out of money or until some idiot drops the ball, at which point everything goes to hell pretty damned quickly," i.e., the current state of digital archiving."

My questions are these: Have you spent enough money and time backing up your digital photos to still make the "old-fashioned" contact sheets and negatives "too much work and cost"? Have you lost any digital photos because of failed storage drives or other "glitches"?

If hard silica glass is the future, I hope they test it (as mentioned by other commenters) when someone steps on the glass on the floor (wearing hard-sole shoes) -- or drops something hard on the glass. How does the glass fare if it is dropped on a corner? Accidents do happen.

[Jim Arthur]

Speaking of revolutionary new tech…PBS Frontline aired an interesting episode on AI the other night that’s worth viewing. It’s available for free on their website. Today’s crazy speed and processing power is driving AI and the recent advances in AI’s visual task ability (cameras/sensors) is both interesting and creepy.

While Frontline tends to cast an ominous tone over everything they cover I think there is plenty to be concerned about with AI. It will be interesting to see how AI shakes out over the next decade. It’s just another tool after all and I imagine its impact will depend in large part on those who wield it. This is the part that worries me though. The lack of ethics in business and government these days is worrisome.

[J Williams]

I'd probably own a Z camera if not for the XQD card. $90 for a 32 GB card?

The Z50 uses SD so anyone wanting to carry along one to compliment or backup their Z6/7 needs 2 storage formats.

Every camera I have is SD. My laptop, desktop and almost every PC out there can read one. Why complicate my life?

Sometimes new tech is simply not worth the headache. If old tech is sufficient, cheaper and easier why change?

[Stephen Scharf]

As far as I am concerned, it remains to be seen where CFExpress will take off as a standard format, much as the original Compact Flash format did back in the day.

CFExpress was developed jointly by Sony, Nikon and Sandisk, but Sandisk has yet to put any resources into manufacturing CFExpress cards, and my guess is this is once again due to Sony's Machiavellian IP practices. Where oh where is Sony Beta?

With respect to actual applications requiring CFExpress, I see this format as more suitable for video applications than stills photography, with the recent release of 8K-capable video cameras.

With respect to stills, the main application for CFExpress would be for shooting sports, and despite the view from YouTube video reviewers who are constantly hyping faster frame rates (e.g., 14 FPS, 20FPS and higher) for new cameras, the fact is these guys are amateurs when it comes to shooting professionally accredited sports.

So here's some context for the gang for consideration: 8 FPS is plenty for shooting professional sports, 11 FPS is just icing on the cake. Most pros shoot in this range of frame-rates, and some even turn the frame rates down to 5-6 FPS.

So, 300 MB/sec UHS-II SD or XQD cards are fine for these applications.

Why? Because sports shooters, unlike amateur YouTube reviewers, shoot like a Navy Seal or Delta Operator: 2-3 "taps" on the shutter when capturing peak action. Its their decades of experience that lets them do this with accuracy and precision. There are always use-cases that require faster frame rates or more frames, but these use-cases are typically outliers and non-standard shooting situations.

All that shooting at 14 or 20 FPS means is: a whole lot more frames in your take you have to cull down to a small set of selects to find the one hero shot that will get published. All this culling takes time away from getting your hero shot sent out in time for deadline press.

The other thing that shooting at 14 - 20 FPS, etc. does is...unecessarily burn up shutter life. And believe me, you don't want to have a shutter go south on you in the middle of a big pro event, as happened to me when shooting a MotoGP in 2007. Ask any pro sports shooter, shutter life is typically one of their very top 5 requirements along with....Durability and Reliability.

[Chris Rusbridge]

We keep getting these claims that X or Y storage medium will solve the digital preservation problem because the storage is permanent, or good for 100 years or so, whatever. It's certainly true that longer-lived storage media are better than shorter-lived ones. But in general the problem soon becomes whether you still have the ability to read the storage medium. There's stacks of media lying around that are pretty readable except that we don't have workable devices to read them. The pressure that you mention... the next device will be "mo' bettah" and will replace the earlier device, specially for less mainstream media.

The digital preservation problem isn't really about the storage medium, it's mostly about continuing care and continuing money. It's generally not a problem to move information from one storage medium to another... while you still have access to the old devices... so you have to have someone there to do it at the right time.

[Come to that, I don't currently have a darkroom or a film scanner, so my old "WORA" system isn't exactly accessible either. At least not immediately from here. --Mike]

[Terence Morrissey]

Syylex AG in Villingen Germany in 2011 invented a glass DVD called the GlassMasterDisc that apparently had a lifespan four times longer than the M-Disc. It was terribly expensive at 160 euro and the company went out of business in 2015.

[aizan]

The "A" in WORA stands for "anywhere," not "always"! The term is not used in the field of digital preservation. A glass storage medium might be durable and more disaster-ready, but remember that you need a lot of machinery and software to actually read it that is more precarious. Human-readable media like film is better when it comes to that.

If anyone wants an intro to digital preservation, I recommend listening to "Preserve This Podcast" from the Metropolitan New York Library Council.

https://preservethispodcast.org

[Nick]

The biggest problem to broader adoption of that Project Silica tech is the writing lasers; the kind of femtosecond laser required for permanently writing index changes into glass is gonna be big, expensive, at least a bit fussy, and sorta dangerous. (As is sometimes said among the high-power laser jocks I've worked with: Do not look directly into the laser with your remaining eye!) I think the tech will be a while in getting down to computer-tower size.

That said, it seems like a fantastic archival storage medium, far better than anything else I've ever heard of. I can definitely see it being broadly adopted by people looking to archive huge quantities of data.

[PDLanum]

I have discussed this sort of thing a few times out there on the web. My experience is from two companies that have governments requiring access to data used to design "things". (Airframes and Health Care devices)
At the airframe company, we had to reload some 18bpi tape. Once the company found a tape drive and got it up and running they mounted the old tapes. They did not snap, as expected, but the oxide separated from the tape in a cloud of brown dust.
This same company was using Optical drives (glass substrate) that looked like CD's in their attempt to have manuals stored around the world. A master set in the US, Australia and somewhere in Europe. I don't know if they are still there, but it was when I left in 1999.
Now, here is the question for interfaces. Will computers be able to read ATA drives, USB drives (some company's have the "courage" to stop using USB ports), SATA, SAS or what ever flavor of the week is popular today. I just went through my old CD's from the early 2000's and they still worked, I am not sure about the ZIP drive and disks I have on the shelf.

[Scott]

The nice “feature” of using rotating hard drives for archival storage is that they usually fail pretty quickly - i.e., a single-digit number of years. If you’re keeping dual archival copies, you then create a new archive using then-current media from your surviving archive. This automatically (lol) ensures that your archive remains readable using current technology.