9

Backups are one thing, but long term archival is another. For example, you might be required to store emails for 7 years, or keep all project data indefinitely. I used to save archives to tape, but then I've had tapes get destroyed (drives rip the tape out). So...write to 2 tapes I hear you say. Is that what others do? Have 2 (or more) tapes of the same data for redundancy?

But then the other issue is that tapes cannot usually be read by different backup software vendors. Eg if you go from Arcserve -> Backup Exec -> Commvault over 10 years you would need to keep all 3 systems so that you could restore old data. Likewise for hardware. Old tapes might not be barcoded. Might not be compatible with the new library etc etc. So do you keep old tape hardware AND old software just in case you might need to restore a 10 year-old file?

Or...when you move to a new backup system do you migrate all archived data to the new system and re-archive it onto new tapes? That could be a huge job.

Any thoughts?

1
  • How much data are you looking to archive?
    – GreenKiwi
    May 21, 2009 at 7:09

6 Answers 6

3

How much data are we talking about? Our "archive" data is small enough that we just keep it in live storage (on a NAS unit) which gets backed up with the usual live data, so it exists just as our usual data does and is subject to the same recovery techniques without having to worry about keeping decade-old technology around. If our live data moves to a new storage platform, the archive moves right along with it. We also set the permissions for archive data such that only a member of the archive security group (of which there are very few member users) have access to remove anything from those folders.

If you're dealing with more data than can reasonably kept in live storage, tape may be your best bet for long-term storage, coupled with keeping at least two copies in different controlled storage locations. Bring them out every couple of years to make sure the media is still readable, and migrate from older media to new media if you're switching vendors to ensure you can continue to access the data.

6
  • 1
    Archives are about 3 or 4TB. This is too much to backup as part of regular backup, it would require many extra tapes each week which is a waste as it never changes. And we don't have spare SAN storage anyway. May 21, 2009 at 2:43
  • 1
    For 3-4TB, I'd pick up a bunch of 1.0-1.5TB external drives and make two sets of backup copies directly on the drives. Seagate makes an enclosure that will take 4 1TB SATA drives and allow access over a single USB connection. You could load up two of those and put them each at different locations. Still bring them out every year or two to ensure they still work and replace drives as needed. Depending on your vendor, tapes may be less expensive. May 21, 2009 at 3:06
  • Yes I guess that is a plausible solution these in these days of cheap disk. I would love to move away from tape, it is just too unreliable (CRC errors, broken tape, label errors etc). May 21, 2009 at 3:24
  • Yeah, I'd go with this option. Drive space is so cheap these days, keeping the data live in a redundant system is the way to go.
    – GreenKiwi
    May 21, 2009 at 7:12
  • 1
    A 4TB backup over usb would take almost 20 hours to complete. Do you not have a window in which to complete the job or, like you said, does you data never change? If you do have a window I would go with something with a higher data rate.
    – JohnyD
    Dec 23, 2009 at 12:03
3

In my case, we do archives to tape, and I will tell you why it makes sense for us.

First of all, the software we use, Tivoli Storage Manager, is hands down, the software we will be using for the foreseeable future. In my opinion TSM is the top of the food chain in terms of enterprise backup software, so the problem of changing to a different software platform is relatively non-existent for us.

That being said, we do maintain quite a few long term archives, and we have a few different ways of doing it. Fortunately for us, we use some of the latest tape technologies which can store upwards of 1TB per cartridge, so having a large quantity of archives live inside the library is not a problem.

  • Archive to local tapes, and copy those archives to our alternate data center, with the archive tapes sitting inside the tape library.

We do this for certain types of data because the data is restored from on a regular basis, and is important enough to require two copies.

  • Archive to local tapes, but not remote tapes, and leave the tapes inside the library.

We do this for slightly less important data, which needs to be restored from on a somewhat regular basis.

  • Archive to local tape, and send them offsite for storage.

The above practices have worked out for us pretty well in the past. We did migrate over to a totally different tape technology a couple years ago, and it was a bit of work to move the data across, but it really wasn't a big deal. In TSM it was literally one command per server worth of archives, and it would run. Might take a couple days to run, but it was not an active work effort for anyone.

The wave of the future is obviously disk based storage though.

As soon as the opportunity comes along, I will be collocating a disk array in a secure storage facility and copying long term audit type archives to a device like that.

2
  • Good ideas. We actually have a similar setup. We have 2 remote facilities with tape libraries. We use Commvault, similar to TSM I guess. The thing is, how do you define "slightly less important data". It's important to someone! And it could be business-critical, unbeknownst to you. May 21, 2009 at 3:28
  • On the disk array, it's worth looking at ZFS on Solaris, or NetApp, which regularly verify per-block checksums, greatly reducing the chance of bit-rot. Any archive approach that doesn't take account of bit-rot seems deficient to me.
    – RichVel
    Jul 12, 2011 at 7:51
1

Two EMC Centera installations (mirrored from different locations). It costs big dollars, but it works.

http://www.emc.com/products/family/emc-centera-family.htm

1
  • That's one up from Avamar isn't it? And we couldn't even afford that. May 21, 2009 at 7:52
0

You could also look at a solution like Data Domain (now NetApp). They archive and do advanced compression that they call DeDupe whereby they look for similar chucks of data and get very high compression ratios.

What sort of data are you trying to back up? Is it all "random" data like video or music? Or is it data that might lend itself to compression?

1
  • Suspect it would cost too much, like Avamar. We use Commvault software which also does DeDupe if you spend the dollars, which we don't have. Damn you GFC! May 21, 2009 at 7:53
0

Move forward with the new archiving solution and keep the tapes for as long as you require that data. If you require it permanently then you might consider migrating it to disk. With disk backups migrating to future media is decidely easier.

A year or maybe 2 from now you can begin disgarding tapes if those backups are no longer required.

Reimplementing is a huge endeavor if time will solve your problem for you. Do the cost analysis if you have a cost associated with storing the backups and/or the software licenses to work with those.

0

Your problem with data archiving, to simplify your enigma, seems to be future compatibility of old tapes with new hardware.

My Solution: choose only one vendor with a complete solution (software and hardware) which you trust will do its best to offer legacy compatibility.

And obviously get a very good price-wise contract considering your fidelity ;)

Not the answer you're looking for? Browse other questions tagged .