Kingston Technology to the Rescue!

Following our multi-part discussion of flash memory cards for cameras kicked off by Ctein last Wednesday, I've been seeking a reasonably authoritative answer to the question "Should you 'Erase All' or should you reformat every time to clear old images and refresh a previously-used card?"

My feeling from surfing about is that there's a lot of opinion ricocheting around and not much from substantial sources.

I might have failed in my efforts to get in touch with SanDisk, whose cards I've used for years, but I got an answer from Kingston Technology, a U.S.-based multinational. Wikipedia says that according to Gartner Inc., the information technology research and advisory company, Kingston is the #1 manufacturer of USB drives and the #3 maker of flash memory cards (although a quick scan of our usual suppliers, Amazon and B&H Photo, indicates that Kingston's lower-level cards are more readily available than its premier products).

Kingston's premium memory card

I put the question like this: "Assuming a card (which has initially been properly formatted for the device) will be reused again and again in the same device (camera), what's the best way to erase the old images before reusing the card? [...] We're trying to establish 'best practices' that photographers should follow to optimize the reliability, dependability, and longevity of their flash memory cards."

This response came from Kingston Flash Engineer Tung Lam:

Most camera manufacturers follow the "Design rule for Camera File System" standard which specifies both directory and file structure for recording data to your device. Typically, when you format a card in your camera (or other device), it will follow this specification so that the file structure produced is exactly what the manufacturer expects. If you've seen a "DCIM" folder on your memory card before, this is the specification in action. Kingston always recommends that photographers format their memory cards within the device they intend to use the card in. This helps to ensure that the card is formatted correctly, will communicate with the device in the manner in which the manufacturer intended and helps prevent data corruption.

As a general rule, there are two options for formatting. A "format" is sufficient to clean the File Allocation Table (FAT) and refresh the card for subsequent use. A "low-level" format (not present on all devices) is designed to prevent users from recovering data in instances where one would want to dispose of the memory card. This approach could take a bit longer depending on the capacity of the card.

Followed by this from David Leong, Kingston's Public Relations Manager:

As a general best practice, Kingston recommends the user format the card.

When you select "erase," the data (e.g. an image) is still there. What is being deleted is the location of the photo stored in the File Allocation Table. "Erase All" deletes every single location in the File Allocation Table. Although the information in the FAT table has been erased, there can still be "residue" from the file. What can happen is the next time something is written in that spot, the controller on the card may decide to skip that spot or write a partial piece of data to it, while the rest of the data is stored elsewhere. This is called fragmentation. Similar to a hard-disk drive, fragmentation can cause performance issues. A "reformat" deletes the entire table and then creates a new one for the next use, meaning that as new information is written to the card, it is done sequentially in the FAT table.

So there's that. Looks like this one company's answer is that it's best to reformat every time. If I get answers from other companies I'll post them.

My thanks to Tung Lam and David Leong of Kingston Technology.

Mike

Original contents copyright 2013 by Michael C. Johnston and/or the bylined author. All Rights Reserved. Links in this post may be to our affiliates; sales through affiliate links may benefit this site.

(To see all the comments, click on the "Comments" link below.)
Featured Comments from:

Bob: "Oh good...I have been doing something correctly from day one (2000) of my digital camera experience to the present. That is not usually how it works for me."

Comments

[Manfred]

I think in the early days some camera vendors had trouble to properly implement the file system operations. E.g. deleting individual files or folders could lead to corrupt file systems. Never doing such operations was good advice at the time. I still tend to erase by doing in-camera formats.

[Hernan Zenteno]

Exactly. I learned this the hard way. I didn't formatted a card in the old times (If I not remember bad was a Nikon F5 with digital sensor from kodak). I erased all the photos. I did my work for the newspaper and when I tried to download the pics I found that the photos where not there, only thumbnails size photos. Since this date I format my cards in camera. Sometimes when a card is working weird/odd I format it in a mac using utility disk. Then I format again the card in the camera I will use. All is ok since that. Well except when a card goes very bad but this is very rare.

[Kenneth Tanaka]

I found it surprising to see how lore-laden the topic of memory card care remains after many years.

Personally, I have been reformatting my cards in-camera for many years and have only had one problem with one card years ago. Good to see the folks at Kingston confirm this as a sound practice.

And thank you, Kingston, for taking time to answer the question for TOP readers! We'll all buy only Kingston cards henceforth!

[Tom Kwas]

Plus One for Kingston, I was always told that you format in the device you want to use, and reformat to erase, in the device, but I have to say, I also own more Kingston cards than San Disc and Lexar, it's always been my high-end "value" card, and they've all worked great. I generally use only San Disc, Lexar, a few Toshibas, and Kingston, and I'm really happy with Kingston, so I'm happy they stepped up and answered your question too!

[MHMG]

I would also add that reformatting the card upon re-insertion even into the same camera can take care of performance issues caused by your computer if, like me, you upload card data to your computer with a card reader rather than from a direct camera connection. For example, if I accidentally remove the card after upload but forget to eject it "safely" from my Mac, my Fuji X100 then becomes extremely sluggish to turn on and be ready to shoot. I must wait a minute or two in order to gain access to the menu where reformatting the card then fixes what's ailing the card as far as my X100 is concerned!

One downside for some amateurs, of course, is that image file recovery programs have the best shot at retrieving accidentally erased image files if the card has not been reformatted. The obvious answer to avoid this unpleasant situation is to create a very rigid file management workflow that backs up your latest image files in duplicate or triplicate before you dare go near that "format card" menu on your camera.

cheers,
Mark

[JPH]

Interesting to learn.

What would be nice then, is if camera manufacturers would take this into account with their products, and change their cameras "delete" functions, to better reflect the best practices of actually deleting digital images files, and to insure that future images captured with their cameras are not potentially corrupted by their cameras writing over fragmented residual data on cards.

Maybe kind of like Apple does in OSX, where you have 'Empty Trash' that "deletes" content, but not really, and 'Secure Empty Trash' that completely deletes the content from disk.

[bill vann]

What still need be addressed is permanence of data,how many years and is it constan hweteher used 1x or 1 million x.

What circumstances can impact the stability of the stored files, e.g. washing machine accident (typically not an issue if no an old microdrive, but best not tested with anything of value, microwave, strong magnetic sources and how strong.

And finally number of write/erase cycles. SSD drives in computers have a finite life, I think for a camer card this is not so critical, but early SSD users usking poor practices could "wear out" their SSD devices in a matter of months.

Essentially I am concered with permanence, failure and its causality. I mean my trite crap needs to be preserved!!

b

b

[Ben Rosengart]

Kingston's advice makes sense to me as an IT nerd, for what it's worth.

[Gato]

Thanks, Mike. I had been waiting to see what you came up with on this. It is surprising how difficult it can be to chase down authoritative answer on some of these basic questions.

It would still be nice to see something on card life and data permanence. I remember seeing numbers many years ago for CF cards and thinking at the time it would take most photographers several hundred years to 'wear out' a card. Have no idea if that still applies, or even if it was correct at the time.

[John Krill]

I worked in the computer industry for 30 years starting in '73. Formating disks was a big deal. From reading the info from Kingston it appears that the actual data is not erased. They may clear out the FAT but that's all. That really isn't a true format. It sounds like a format of a memory card is just a glorified delete all. Remember these cards are treated like a disk drive and divided into sectors. Each sector has an address block and a data block. A true format would re-write the address block and write some special pattern into the data block. As end users we have no way of doing a low level format. What should happen is the FAT is cleared and special data should be written into each sector. Doesn't sound like they do that.

So for me it doesn't matter. Erase or format, same difference.

[ctein]

Dear MHMG,

That may be true for a “shallow” recovery scan , but it won't be for a deep one. Shallow recoveries just try to recover directory information, and that frequently works to get back your images. Totally erasing that information, as in a format, makes it impossible. A deep recovery scan actually reads all the data blocks on the card and reconstructs the files from that. It's not dependent on any pre-existing directory information.

A deep scan is likely to recover MORE images on a card that's been reformatted and then partially overwritten with new photographs, because the new information is being written in a less scattershot fashion across all the blocks (less fragmentation).

The program I mentioned using in my column, Data Rescue 3, can do shallow or deep scans. I just tested it on a formatted card and the deep scan worked perfectly (shallow, of course, came up with bupkis).

How can you tell if your data recovery program does shallow or deep scans? If the scan takes only a second or so per gigabyte, then it's not doing a deep scan. Obvious if you think about it-- there's no way to read that much data that fast from the drive. If the software tells you to go make a cup of tea, read the morning paper and then come back and check on it, it's doing a deep scan.

With Data Rescue and the 16 GB card, for example, the Quick Scan took barely 5 seconds on my iMac. The Deep Scan took 25 min. to read and analyze the card.

As a rule, heavy-duty software that can do deep scans costs a lot more than the lightweight stuff (some of which is free). Whether you want to invest the money depends on how traumatic it is to lose some photographs that you might be able to recover by throwing more money at them.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Ger Lawlor]

Hi Mike. Following fhis with interest. FYI, It is possible to engineer a major disaster with multiple cards, involving swapping them while an image programme such as capture one is looking at the card, and then closing the program. Result: file allocation table for card A gets written to card B. been there, got the T-shirt, but sandisc rescue utility saved my bacon. I keep all my CF cards in a rigid holder ( holds 4). As I fill each card, I replace it in the same holder, but with the back facing up. Foolproof, and i can judge how much capacity remains when on a job. I see some pros putting cards into pockets or throwing them into bags as they fill.... Makes me very uneasy. To ensure no accidents, i never delete onsite. Before every shoot, I format every card that I'm taking with me, and keep one emergency card in the car for the day I forget to bring any with me. Finally, cards never get left in the camera after a shoot. The card at that point is worth an awful lot more than the camera...

[Lukasz Kubica]

This is nothing new and nothing specific to memory cards. FAT file system is an old "friend" from DOS and Windows 95/98 times. It's primary advantage is simplicity (i.e. it's easy to implement for camera vendors) thus it is a standard for USB pendrives and cards. It is however prone to data loss when power is switched off during writing. It is also prone to fragmentation, but this is not really an issue with cards and I would like to see a benchmark which proves a significant performance penalty here. It was true for hard discs, because HD is a mechanical device and accessing lots of data blocks scattered around required lots of mechanical activity which is always many times slower than simply reading things from random access memory.
So to sum up, quick format will prevent fragmentation and your card will be a bit faster, but it can be so little that it could be negligible.

[Mike Plews]

We have sixteen digital tv cameras where I work, each holding two 16GB cards and it is standard procedure to format all cards at the end of your shift.
Fours years later, no problems. FYI these are a combination of SxS, SDHC Class 10 and Sony Memory Stick and no problems associated with reformatting at all. We do this via the camera menu.

[psu]

As I had said before, at the lowest level of the file system there isn't much difference between a format and erase-all. All the bits are still there in any case.

Formatting is a simpler operation though (just erase the whole table) and is therefore less likely to go wrong. That's why you should do it.

[Rory O'Toole]

Yes, but did your kindle turn up, or have you had to resort to pulp?

[I was forced to conclude either that my Kindle was stolen, or I left it some place. (Sadly the latter is more likely these days.) The new Kindle Paperwhite arrived today. Review forthcoming. --Mike]

[Marty]

Well, Wilco Tango Foxtrot.
After that previous post I bought that very same card through from the TOP lynx.
how 'bout that?

[Ernest Bernardo]

I like this stuff. I am a computer programmer, old, who predates MS-DOS and PCs by 25 years when I started with computers.

FAT file systems have 3 main components - a directory containing filenames, a File Allocation Table (FAT), and a data area containing clusters (in turn comprised of sectors (nominally 512 characters each)).

The FAT maps the allocation of clusters. A cluster can only associate with one file.

Either deleting files or formatting the file system clears the directory entry with the file name and marks the allocation chain of clusters as newly available.

Camera FAT file systems vary from ones on the computer in that there is one copy of the FAT, not two, and the clusters contain more sectors than those typically used on computers.

The reason for this is the camera writes a sequential series of fairly large files. Using large clusters minimizes the processing necessary to allocate additional clusters per file. Large clusters are more efficient for large files and an additional benefit is a smaller FAT table, leaving some additional space for images.

I've formatted camera cards in cameras and also on computers and have used them interchangeably in both devices to no ill effect. The camera takes a noticeable amount of extra time to write to a computer formatted card.

Fragmentation can only occur when some images are erased on the card and others left, creating empty spaces of varying sizes. This could result in clusters of the same image not being contiguous.

Having done this over many years with various cameras, some are better at dealing with legitimate FAT variables than others.

My best practice - format cards in the camera used to take pictures, don't erase individual images unless really pinched for space.

Once formatted in camera, I've never seen a problem on any camera when deleting all photos whether done on a computer or in camera.

For the sake of speed, I delete all photos on a card via computer, I format a card via the camera.

It is likely that neither deleting nor formatting a camera card really removes the images in the data area. There are recovery programs that prove this out.

"Low level formatting" can accomplish such removal by overwriting the entire data area. Such low level format facility is not easily found for either cameras or computers.

My two cents.

Thanks

[Oskar Ojala]

Lam's answer makes perfect sense. Note that here formatting is a risk-minimizing strategy used when the same card is used in multiple devices; using the card always in the same device and deleting isn't a big deal.

Leong's answer however doesn't make sense to me; the card doesn't know about the file system and the host doesn't know about which flash cells the data is actually stored in. Flash devices use a technique called wear-leveling, which means that controller (in the card) tries to store the data equally to all parts of the card, so that no flash cell wears out prematurely. Thus, fragmentation doesn't apply to flash devices, since the data is fragmented by design. Now if Kingston doesn't do wear leveling then I'm happy to keep using Sandisk cards ;-)

Personally I format in camera because it's convenient and Nikon even provides a quick key combo for it. I do delete bad shots in the camera with no second thoughts. I'm a software professional with experience in operating systems.

Actually this whole discussion is a bit akin to whether to user protective filters or not...

[Steve Jacob]

This is all pretty good advice, but there were a couple of interesting questions about longevity of solid state memory.

Firstly, for anyone concerned, solid state memory is electronic and pretty much immune to magnetic fields. They are also pretty tolerant of ionising radiation so you should not have any issues with airport scanners.

Microwaves are not ionising and will have no effect, but like all metal-containing objects it's best not to put one in your microwave....

Secondly, finite lifespans. It's true that cards can only support a finite number of state changes per bit - but the number is very substantial and this only occurs during write operations. If you used an SSD as a swap drive it would wear out faster than if you used it as a boot/OS drive, but it is still likely to last longer than an HDD.

For use in cameras each location will be written to at most once between each format unless you delete files in the camera (in which case the location may be reused if short of space). This is a very low level of write activity and will have no real effect on longevity.

They do fail for much more boring reasons. Voltage spikes and static, bent pins on CF cards, wear or corrosion of SD contacts and mechanical or heat stress. They are not physically indestructible and contacts on low voltage devices are a weak spot.

They can also get messed up if you whip them out of a computer during an I/O operation. However this is not a hardware failure and you can normally reformat them in camera (though you will lose the data).

Overall, if you buy from a reputable manufacturer most cards will last a very long time if handled properly, or fail relatively quickly because of a defect in the card or the card reader.

[Nicolas Woollaston]

Some flash memory can be good for as few as 1000 write cycles (per bit or byte or block). Internal circuitry tries to spread the usage around so that it all wears out evenly. I am not sure how it is typically implemented in cameras, but low level formatting that rewrites the entire "surface" may count as part of your 1000 cycles and so may not be helpful if you want your card to last as long as possible. Can anyone who has delved into the technical details comment on this.

[phil martin]

which is what I've always done, good

[Eric Perlberg]

There was a very interesting talk about the workings of SD cards on the now defunct rangefindermoment podcast here http://rangefindermoment.com/rangefinder-moment-podcast/2012/5/7/rm005-interview-with-scott-a-moulton-deep-dive-into-the-worl.html
Although you'd think the subject dry, it wasn't. This issue and others we commonly run into as photographers are discussed.

[Steve Jacob]

@Nicolas

I think commercially available flash cards are rated at around 100,000 cycles - way more than you will need for camera use.

Server SSD drives are rated higher than that. Up to a million I believe.

Over very long periods however the information on a card can decay if not reset, because of interference from neighbouring cells - means they are not good for archive storage. We are talking years however not months.

[Eric Perlberg]

Mike, unfortunately the rangefindermoment podcast seems to no longer be available either from the linked page or through podcast catchers. Apologies.

[Thom Hogan]

Ernest has it basically correct, though I want to point out a couple of additional things that's important to this discussion.

All the early digital cameras used variants of real time DOS (most still do), thus the use of FAT in the DCF specifications. Remember that FAT itself went through changes (FAT16, FAT32, etc.), and despite the DCF standard there was still enough slop to let makers do things differently (single FAT versus FAT backup, etc.).

Where a lot of people get hung up without noticing is that the camera is using one definition of FAT while the default format on the OS they're using is a different one. This was a big problem back when a lot of cameras were FAT16 but Microsoft had switched to FAT32.

But the real thing you need to know is this: a camera format and the default OS formats don't look for and mark "bad blocks." A true low-level format does. Thus, if a card develops a sector problem, unless you perform a low-level format, that problem is going to remain; a camera format won't remove it.

[Val]

I 'Format' my CF and SD cards on my PC before I use them in the Camera.
The Camera 'format' clears the FAT Table but does NOT erase the images from the card.
The PC 'Format' erases all information from the CD and SD cards.
Should you want to 'test' this method, take a CF/SD card that has images on it, and do the Camera 'format'. Then run the SanDisk or Lexar recover programs.
Repeat the test but this time 'Format' the CF/SD card on your PC. Run the SanDisk or Lexar recover programs again and compare the results.

[Brian Miller]

Fragmentation is not an issue with flash cards, such as SD or CF. The time to access the first block is the same time to access the last block, because both are a memory location on the card. However, on a spinning hard drive, the head must physically travel from track to track, so it takes more time to read a file when the files are fragmented on the drive. Thus, hard drives should be periodically defragmented to speed up file access.

[Pak-Ming Wan]

Ok, so I'll suck up and say I'm wrong. The fragmentation issue on a flash card used in a camera isn't likely to be the fragmentation of the filesystem, is the fragmentation of the actual cells on the flash drive. But if Kingston says that for FAT/FAT32 based filesystems the best way to avoid this is a reformat, I'll go with that.

I'm guessing it is because writing to the relatively small FAT header at the start of the disk will result in smallish allocation of new blocks on the flash drive at a physical level by the wear levelling algorithm, which has a higher probability of fragmentation, whereas a reformat just resets the wear levelling algorithm all together. I just wish someone had a better explaination for all of this.

On the drive recovery tools: try doing that on an SSD / flash drive which has had a controller fail. You lose everything, immediately. It's a failure scenario which is much less likely on a hard drive, and something to keep in mind ...

Pak