I've been thinking about data integrity, I currently backup about 2tb of data and always have one backup of data from a year ago.

My concern is if a file became corrupt on our production file server no one would notice because some files aren't accessed for many years and if corruption occurred I'd be backing up a corrupted file.

I'm not sure how I should handle this problem, is there a way to detect data corruption? Or is the only solution to store older backups in case something becomes corrupted and isn't noticed?

| improve this question | | | | |

In my experience, each file type needs its own checks to determine if something is indeed corrupt. Data is just dots and dashes at its heart, and what determines "corruption" is entirely file dependent. You will need to determine what file types are most important, and then determine if it is reasonably possible to create automation that checks for the file type's consistency. That will be a daunting task as file type specifications change over time and as you encounter proprietary formats that have no easy way to programmatically determing corruption.

Furthermore, data corruption is only a part of the problem. Sometimes files can be wrong from a human perspective, but consistent from a data structure perspective. If someone mistakenly edits a file - the data is fine from a corruption standpoint.

Ultimately you need to sit down with the leadership of the business and determine what the most important data assets are for the company. Then determine how long those need to be retained and with what level of recall. Do they want fine-graned point-in-time recovery to four years in the past? Maybe only for certain files but not for others?

Considering that you only have 2TB to backup, a GFS tape backup scheme using LTO4 cartridges can allow you to reach back many years with relatively few tapes. This is, of course, entirely depdendent on data churn. If you have a lot of busy bits, then you'll have more tapes. Still, 2TB is a relative speck that LTO4 or even commodity disk storage would yawn at to keep a few years of data around.

It's not an easy task to protect digital assets. Keep the Tums handy.

| improve this answer | | | | |
  • +1 to mention user error causing bad data to be backed up – Bigbio2002 Feb 23 '12 at 21:05

This is one of the reasons why ZFS is so great, no more data corruption. Sharing out ZFS filesystems over CIFS is pretty straightforward.

| improve this answer | | | | |
  • This seems like the solution to my problem and that wiki was a great read. Shame we're currently on Windows and won't be able to move for a while. Also the term "silent corruption" has given methe term I need to Google for to learn more. The only bit that confused me was "ZFS cannot fully protect the user's data when using a hardware RAID controller". So I presume this means it only works with software RAID as I just read a little about its support for Raid-Z? – Sean Bannister Feb 20 '12 at 14:24
  • In my search I just read about ReFS coming to Windows 8 Server, don't know much about it yet but for anyone searching it might be interesting to look into. – Sean Bannister Feb 20 '12 at 14:33
  • ZFS is "software raid", yes, but is superior to hardware raid controllers in features & functionality. The server itself becomes the hardware RAID controller. The only downside is you need a big, powerful server for ZFS to take advantage of some of it's features. For data protection (no corruption) alone, you can get by with relatively little. I've seen white papers on ReFS & it's a good thing. ZFS will still be superior. – churnd Feb 20 '12 at 15:59
  • 1
    You cannot assume that data is never corrupted above the filesystem layer. The most obvious example is someone simply writing incorrect data into the file. – wfaulk Feb 20 '12 at 18:51
  • What if the application smashes a stack and puts out a bad data stream? It will have consistent writes to the filesystem, but the data structure of the file itself will be bad. I thought the OP was referring to checking the consistency of individual file types, and not just if there had been a consistent write to the filesystem. – Wesley Feb 20 '12 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.