One has to be particularly careful when one desires long term storage of photos. I try to use a multiple solution strategy. I back up photos on more than one hard drive and I burn multiple copies onto discs. Both solutions can create problems though. When labeling a disc one should choose their method carefully. Marker pens can corrupt data over time, as can sticker label adhesives. It is best to use a quality labeling method. I use the DiscPainter by Dymo and have had no data corruptions issues at all. It prints color labels along with texts descriptions of the disc's contents in just one minute. Kodak is coming out with a disc that they claim will keep data stable for 100 years! It should be ready sometime in 2008. I like using discs to back up photos because hard drives are magnetic and all sorts of things can corrupt magnetically stored data over time. As for online back up companies, do you really want to depend on a company being around 5, 10, 20 years from now? They may be useful as part of your overall back up plan but they should not be the only method you depend on.
-
Join the MyBroadband community
South Africa’s biggest forum. Discuss, discover, and connect with thousands of members.
-
Question of the Day: How do you normally pay for purchases in stores?
Digital Photos - long term storage concerns
- Thread starter koffiejunkie
- Start date
ld13
Honorary Master
What about those Kodak/Memorex "Archival" discs with a layer of 24-carat gold to prevent oxidation that are designed to last 300 years. 
Normal HDD works great for me too though. (I don't have a camera, thus I don't have a Big collection, yet)
Normal HDD works great for me too though. (I don't have a camera, thus I don't have a Big collection, yet)
koffiejunkie
Executive Member
Doesn't really matter how many layers of what the plaster onto the discs - they're still made of plastic.
As I've watched my personal photo collection grow over the last two years, something started bothering me. How future proof is storing images on a hard disc/flash card/CD/DVD?
Hard discs fail. Flash cards fail. CD/DVD fail despite their theoretical 100 year lifespan. OK, so we use multiple backups on different medial. But how sustainable is this? Cameras are making bigger images all the time. What do we do when our collections reach a TB and over?
DVD-R media has a theoretical lifespan of 100 years. So far I've had no DVDRs or CD-Rs fail for reasons of their own, I've only had cheap CDR's fail when someone used a permanent marker on them which at through the lacquer and damaged the CD from the inside. Those CDs were very cheap and are about 10 years old now and I make sure to only use certified DVD/CD markers nowadays.
Your best bet for storage on DVD is to use DVD+R format. It offers increased redundancy and error recovery vs DVD-R. You should also store data with additional PAR
(parity) files - there is a utility available which generates these large files,
so your storage capacity diminishes but if something gets corrupted the
PARity file can usually recover the lost bytes.
Make - the BEST DVD-R's out there are single sided Taiyo Yuden media.
Second best is Verbatim. Its difficult to get original TY's in SA - they sell under "That's" in Japan and Taiyo Yuden in the States but if you can get genuine TY's those are the ones to go with. Otherwise Verbatim are almost as good (even rebranded TYs occasionally). For CDRs there was a time when Kodak produced Kodak Gold CD-Rs they haven't done so for years.
Anyway use the best quality media, store it properly, use Parity files
and keep a set of 2 in 2 different geographical locations, mail a set to
your friends / family in another town or abroad.
Storage format - RAW is probably the best but huge. You can also store
in JPEG but it's lossy - but JPEG has been around for years and shows no
signs of dying.
http://en.wikipedia.org/wiki/Taiyo_Yuden
http://en.wikipedia.org/wiki/Par_Files
Frequent verification of your optical media should allow you to foresee
sudden failures well in advance as media shouldn't all fail at once.
ld13
Honorary Master
I did not know that...I know +R has some hardware based recovery measures available but afaik they've not been implemented yet.
They stopped making them? Are you sure? Because I can still buy the 'Kodak Gold Preservation Archival Grade' DVD's & CD's
There have been a number of articles on the web commenting on the way data is recorded on DVD+Rs. The method you mention is part of the DVD+R specification and so has been implemented from the start, its not a reading method but the way the data is written which includes increased redundancy. The details elude my memory though.
I think if you do a search on this forum you'll probably even find a link to one or the whole text pasted in.
I was aware that Kodak stopped making those. If you can get them go ahead, if you can't get TY's get Kodak media. I haven't been able to find Kodak gold for ages, in fact I bought my last batch in 2001.
I think the best bet is to use top graded media like Taiyo Yuden or Kodak,
use DVD+R and use Parity files, sure you waste 10-20% of the space, but
error recovery is improved. You could forgo parity but instead keep two copies of each file on each DVD but then Parity files allow you to reconstruct files while two copies won't allow you that if both are corrupted.
Why DVD+R?
This is the most technical section of the article. If you don’t understand the basics of how CD/DVD media works, or find such technical discussions boring, skip to the next section.
As I said earlier, DVD-R sucks for data preservation for three reasons: inferior error correction, inferior ‘wobble’ tracking, and the fact its data writing methods look like an un-needed halfway point between CD-R and DVD+R. The wobble tracking I shall explain first, then the error corrections method, then the specifics of ATIP/pre-pit/ADIP optimum power settings.
For a CD/DVD burner to track where it is on the disc, it uses three things: the ‘wobble’ of the data track (where it actually wobbles back and forth instead of in a straight line) to tell where it is in the track, the position of the track to tell where it is on the disc, and some additional information on the disc to tell where the track (singular, as CDs and DVDs only have one track, and it is written in a concentric spiral) begins and ends.
This additional information on a CD-R is called the ATIP (Absolute Time In Pregroove), which contains how long the track is, where it begins, what the maximum and minimum writing speeds are, what formula dye it uses, who actually made it, optimum power control settings, and error correction data. The ATIP is stored as a frequency modulation in the wobble itself.
However, since the wobble changes subtly to encode data, it is impossible to use with the small size of tracks DVD requires, as electric noise in the laser pickup and wobbles introduced by the electric motor spinning the disc, these could easily be read as frequency changes in the real track itself.
On DVD-R, they tried to solve the problem with something called ‘pre-pits’ where spikes in the amplitude of the wobble appear due to pits fully out of phase with the rest of the track (ie, between two spirals of the track, where there is no data). This can be viewed as a simple improvement over CD-R as it makes it easier to track the wobble (since the wobble is constant except for the easy to detect and remove spikes).
Unfortunately, this method as one flaw: due to electric noise in the laser pickup, it would be very easy to miss the pre-pit (or read one that wasn’t actually there) if the disc were damaged or spun at fast speeds. The time to read a pre-pit is 1T (roughly .0000000038th of a second), which even for a computer can be easy to miss. DVD-R traded hard to track frequency changes for hard to read wobble-encoded data.
On a DVD+R, however, they came up with a much better method. Instead of changing the frequency of the wobble, or causing amplitude spikes in the wobble, they use complete phase changes. Where CD-R’s and DVD-R’s methods make you choose between either easy wobble tracking or easy ATIP reading, DVD+R’s method makes it very easy to track the wobble, and also very easy to encode data into the wobble. DVD+R’s method is called ADIP (ADdress In Pre-groove), which uses a phase change method.
With ADIPs’ phase changes, the direction of the wobble changes and continues on going in the exact opposite direction (ie, counter-clockwise to clockwise, or the reverse). For example, if the wobble was ‘going up’, the phase change causes it to instantly reverse direction start ‘going down’ no matter where it in the wobble cycle. The phase change is very easy to detect, and also continues for a set period (in this case, one 32T section of the track, or 32 times longer than the pre-pit method of DVD-R).
The state of the phase change (clockwise or counter-clockwise) encodes the individual bits in each block In essence, with the phase change method, not only do you have an easy way of tracking the wobble, but you now have an easy way of reading wobble-encoded data.
As I mentioned earlier, this wobble-encoded data includes error correction of wobble-encoded data itself. Error correction is the most important part of media, because if it does not work, then you’ve lost your data, even if there is nothing seriously wrong with the disc.
The DVD-R specification states that for every 192 bits, 64 of them are not protected under any scheme, 24 of them are protected by 24 bits of parity, and the last 56 bits are protected by another 24 bits of parity. This weird (to put it mildly) scheme allows you to easily scramble or lose 25% of the data that is required to read your disk! This information is almost more important than the actual data burned on the disc itself.
The DVD+R specification, however, states that for every 204 bits of information, it is split into four blocks of 52 bits containing 1 sync bit to prevent misreading because of phase changes, 31 bits of data, and a 20 bit parity (that protects all 32 bits of data). The sync bit is always the same value in all four blocks, and exists only to prevent phase inversions.
Now, the third item on the list: how DVD+R discs burn better. As I said earlier, ATIP/pre-pit/ADIP stores information about optimum power control settings. This information is basically formulas stating how much output power is needed, what the laser startup power should be, and other pieces of information you require to properly burn a DVD.
Optimum power control output is dependent on three things: burning speed, laser wavelength, and information given to the drive about the media. DVD-R basically fails on all three accounts because DVD+R simply includes far more information about the media in the ADIP data than DVD-R does in it’s pre-pit data.
DVD+R includes four optimum profiles, one for four major burning speeds (usually 2x, 4x, 6x, and 8x, though this can change as speeds increase). Each of these profiles include optimum power output based on laser wavelength, more precise laser power settings, and other additional information. With this information, any DVD+R burner can far more optimize it’s burning strategy to fit the media than it can with DVD-R, consistently providing better burns.
For comparison, DVD-R includes one profile, optimum power output based for that one profile only and uncalibrated towards what wavelength it is for, less precise laser power settings, and no other additional information. Typically, DVD-R burners have to already know how to burn a certain piece of media (and include this information in their firmwares) before they can properly burn to it. New media often is not properly supported.
In addition to the optimum power control profiles, DVD+R also gives four times more scratch space for the drive to calibrate the laser on; more space can only improve the calibration quality. So, in short, DVD+R media exists to simply produce better burns and protect your data better.
http://focusfaction.com/forums/showthread.php?t=1457
Finding Taiyo Yuden Media:
http://club.cdfreaks.com/f33/taiyo-yuden-faq-178622/
This is the most technical section of the article. If you don’t understand the basics of how CD/DVD media works, or find such technical discussions boring, skip to the next section.
As I said earlier, DVD-R sucks for data preservation for three reasons: inferior error correction, inferior ‘wobble’ tracking, and the fact its data writing methods look like an un-needed halfway point between CD-R and DVD+R. The wobble tracking I shall explain first, then the error corrections method, then the specifics of ATIP/pre-pit/ADIP optimum power settings.
For a CD/DVD burner to track where it is on the disc, it uses three things: the ‘wobble’ of the data track (where it actually wobbles back and forth instead of in a straight line) to tell where it is in the track, the position of the track to tell where it is on the disc, and some additional information on the disc to tell where the track (singular, as CDs and DVDs only have one track, and it is written in a concentric spiral) begins and ends.
This additional information on a CD-R is called the ATIP (Absolute Time In Pregroove), which contains how long the track is, where it begins, what the maximum and minimum writing speeds are, what formula dye it uses, who actually made it, optimum power control settings, and error correction data. The ATIP is stored as a frequency modulation in the wobble itself.
However, since the wobble changes subtly to encode data, it is impossible to use with the small size of tracks DVD requires, as electric noise in the laser pickup and wobbles introduced by the electric motor spinning the disc, these could easily be read as frequency changes in the real track itself.
On DVD-R, they tried to solve the problem with something called ‘pre-pits’ where spikes in the amplitude of the wobble appear due to pits fully out of phase with the rest of the track (ie, between two spirals of the track, where there is no data). This can be viewed as a simple improvement over CD-R as it makes it easier to track the wobble (since the wobble is constant except for the easy to detect and remove spikes).
Unfortunately, this method as one flaw: due to electric noise in the laser pickup, it would be very easy to miss the pre-pit (or read one that wasn’t actually there) if the disc were damaged or spun at fast speeds. The time to read a pre-pit is 1T (roughly .0000000038th of a second), which even for a computer can be easy to miss. DVD-R traded hard to track frequency changes for hard to read wobble-encoded data.
On a DVD+R, however, they came up with a much better method. Instead of changing the frequency of the wobble, or causing amplitude spikes in the wobble, they use complete phase changes. Where CD-R’s and DVD-R’s methods make you choose between either easy wobble tracking or easy ATIP reading, DVD+R’s method makes it very easy to track the wobble, and also very easy to encode data into the wobble. DVD+R’s method is called ADIP (ADdress In Pre-groove), which uses a phase change method.
With ADIPs’ phase changes, the direction of the wobble changes and continues on going in the exact opposite direction (ie, counter-clockwise to clockwise, or the reverse). For example, if the wobble was ‘going up’, the phase change causes it to instantly reverse direction start ‘going down’ no matter where it in the wobble cycle. The phase change is very easy to detect, and also continues for a set period (in this case, one 32T section of the track, or 32 times longer than the pre-pit method of DVD-R).
The state of the phase change (clockwise or counter-clockwise) encodes the individual bits in each block In essence, with the phase change method, not only do you have an easy way of tracking the wobble, but you now have an easy way of reading wobble-encoded data.
As I mentioned earlier, this wobble-encoded data includes error correction of wobble-encoded data itself. Error correction is the most important part of media, because if it does not work, then you’ve lost your data, even if there is nothing seriously wrong with the disc.
The DVD-R specification states that for every 192 bits, 64 of them are not protected under any scheme, 24 of them are protected by 24 bits of parity, and the last 56 bits are protected by another 24 bits of parity. This weird (to put it mildly) scheme allows you to easily scramble or lose 25% of the data that is required to read your disk! This information is almost more important than the actual data burned on the disc itself.
The DVD+R specification, however, states that for every 204 bits of information, it is split into four blocks of 52 bits containing 1 sync bit to prevent misreading because of phase changes, 31 bits of data, and a 20 bit parity (that protects all 32 bits of data). The sync bit is always the same value in all four blocks, and exists only to prevent phase inversions.
Now, the third item on the list: how DVD+R discs burn better. As I said earlier, ATIP/pre-pit/ADIP stores information about optimum power control settings. This information is basically formulas stating how much output power is needed, what the laser startup power should be, and other pieces of information you require to properly burn a DVD.
Optimum power control output is dependent on three things: burning speed, laser wavelength, and information given to the drive about the media. DVD-R basically fails on all three accounts because DVD+R simply includes far more information about the media in the ADIP data than DVD-R does in it’s pre-pit data.
DVD+R includes four optimum profiles, one for four major burning speeds (usually 2x, 4x, 6x, and 8x, though this can change as speeds increase). Each of these profiles include optimum power output based on laser wavelength, more precise laser power settings, and other additional information. With this information, any DVD+R burner can far more optimize it’s burning strategy to fit the media than it can with DVD-R, consistently providing better burns.
For comparison, DVD-R includes one profile, optimum power output based for that one profile only and uncalibrated towards what wavelength it is for, less precise laser power settings, and no other additional information. Typically, DVD-R burners have to already know how to burn a certain piece of media (and include this information in their firmwares) before they can properly burn to it. New media often is not properly supported.
In addition to the optimum power control profiles, DVD+R also gives four times more scratch space for the drive to calibrate the laser on; more space can only improve the calibration quality. So, in short, DVD+R media exists to simply produce better burns and protect your data better.
http://focusfaction.com/forums/showthread.php?t=1457
Finding Taiyo Yuden Media:
http://club.cdfreaks.com/f33/taiyo-yuden-faq-178622/
Last edited:
schitz011
Expert Member
and all our music on a record while we're at it?
Film has its own special considerations when it comes to storage too right?
Indeed film does have a number of things that can degrade the quality over time.
Firstly it's organic in it's make up, secondly if it's not fixed and washed properly the remaining silver halides will 'rot' and cloud the film. (ever seen BW images that have purple or orange spots on them? Thats rotting halides).
As for digital, having a 3 tier back up (local HDD, external HDD storage and then DVD's) should be sufficient so long as they are in separate locations. In terms of format, while no doubt there will be a new format at some point in the future, just as RAW was an additional new format at one time.
Where as CD's replaced vinyl as defacto standard mainly due to certain quality issues (for the average user before all you audiophiles claim vinyl is superior!
), RAW now does pretty much everything a digital format needs to do i.e transfer all data captured by the camera to a PC/MAC for editing and conversion. While RAW will be improved upon, and indeed is already, I doubt it will be replaced as the ultimate digital imaging format. Although Hasselblad are creating a format which allows for manipulation of focus which is very interesting, but most likely I would imagine this would be incorporated into RAW after they trial it with their own native 3FR format (which is painful if you're not shooting tethered and edit on a machine without Flexcolour, in which case you have to find Flexcolour somewhere and convert out of 3FR to edit the buggers! Or you could just shoot in plain RAW output and lose some space on the card, and at 25MP, cards get filled up quick!)
In fact that is exactly the space where it has thoroughly surpassed it.
koffiejunkie
Executive Member
On what medium? CD is still the only dedicated digital audio medium. It's noise reduction has far surpassed analogue but it has some way to go accurate sound reproduction. Why? Because digital only captures a rather small part of the sound spectrum.
Prometheus
Banned
- Joined
- Mar 19, 2006
- Messages
- 4,252
- Reaction score
- 3
Safest bet is to store primarily on harddrive. Then two copies on permanent media kept in different places - very important. Don't use any form of RW media. Yes it's stated that these can be reused up to 100 times but in practice errors start to develop even after just the 2nd use. RAID is not an effective way of ensuring redundancy. As all the drives are in the same spot physical damage as a result of water, fire, electrical surges, and other catastrophes can just as easily affect the mirror image. RAID with redundancy or parity checking can be a good way of ensuring source integrity, returning to that later.
Flash memory is NOT a good option for backing up or for primary storage. Besides that it's more expensive than other media it doesn't have such good retention. They do sometimes give figures going into hundreds of years of storage but this is assuming power is applied on a regular basis. When no power is applied those trapped "hot" electrons can become untrapped in as little as 10 years or less. I also had a bit get permanently stuck on a new device, in the FAT no less. Flash memory is aptly named and intended just for that, storing in a flash so you can move it to other media.
Another problem I heard of is harddrives becoming corrupt by just sitting there. Someone actually termed this bit rot referring to the bits rotting away in their reality existence. It won't make any difference if you backup to 1 or 1,000 copies when your source is corrupt. This will become apparent when the source fails and the backups suddenly give the same corrupt data. Secondary media usually has a means to examine data integrity and while this is present to an extent in primary media it is expected to be more stable and so used more to detect when the media is going bad rather than detecting random errors. An additional means of parity or crc checking on a file basis is recommended but RAID can also suffice. The argument against this basicaly says nothing can go wrong so we continue backing up faulty data.
I should also mention that these same rules apply equally to passwords and encryption keys. It's one thing to lose some random data but quite a different matter to lose your entire collection or a large chunk of it because you lost your keys. How often have we heard someone losing their keys or passwords and then while at the same time they want it to be uncrackable they also want a means to "reconstruct" it. Take it as a given there is no means to get it back other than not losing it in the first place. In this instance flash memory DOES seem to be the ideal but check and compare each set to ensure its integrity.
I wouldn't worry about media formats. The formats around today will still be around tomorrow and even if in some unforeseeable time into the future they aren't there will still be a means to read and convert them to newer (hopefully) better formats. Asking which format is the best to use is like asking what happens if the sun stops shining tomorrow. Some formats are better than others but the only consideration should be its intended use and purpose.
Flash memory is NOT a good option for backing up or for primary storage. Besides that it's more expensive than other media it doesn't have such good retention. They do sometimes give figures going into hundreds of years of storage but this is assuming power is applied on a regular basis. When no power is applied those trapped "hot" electrons can become untrapped in as little as 10 years or less. I also had a bit get permanently stuck on a new device, in the FAT no less. Flash memory is aptly named and intended just for that, storing in a flash so you can move it to other media.
Another problem I heard of is harddrives becoming corrupt by just sitting there. Someone actually termed this bit rot referring to the bits rotting away in their reality existence. It won't make any difference if you backup to 1 or 1,000 copies when your source is corrupt. This will become apparent when the source fails and the backups suddenly give the same corrupt data. Secondary media usually has a means to examine data integrity and while this is present to an extent in primary media it is expected to be more stable and so used more to detect when the media is going bad rather than detecting random errors. An additional means of parity or crc checking on a file basis is recommended but RAID can also suffice. The argument against this basicaly says nothing can go wrong so we continue backing up faulty data.
I should also mention that these same rules apply equally to passwords and encryption keys. It's one thing to lose some random data but quite a different matter to lose your entire collection or a large chunk of it because you lost your keys. How often have we heard someone losing their keys or passwords and then while at the same time they want it to be uncrackable they also want a means to "reconstruct" it. Take it as a given there is no means to get it back other than not losing it in the first place. In this instance flash memory DOES seem to be the ideal but check and compare each set to ensure its integrity.
I wouldn't worry about media formats. The formats around today will still be around tomorrow and even if in some unforeseeable time into the future they aren't there will still be a means to read and convert them to newer (hopefully) better formats. Asking which format is the best to use is like asking what happens if the sun stops shining tomorrow. Some formats are better than others but the only consideration should be its intended use and purpose.