14

I'm looking at implementing a very large storage server to be used as live NAS for several other servers (all Linux-based).

By very large, I mean between 4TB and 20TB usable space (although it's unlikely we'll actually make it 20TB).

The storage server will be RAID 10 for data security and performance, but we'll still need a backup solution including off-site backup.

My question is: How do you backup that much data!?

It's not like I can just connect a portable hard-drive and transfer the files over. We currently have no other devices with this much storage space.

Do I need to budget for a second, off-site storage server or is there a better solution?

3
  • 5
    I'll leave my usual comment harping about backing being offline. I get really nervous about a backup system being "live and online" all the time. If an attacker can get at your production system and your backups then they can trash your backups right after they finish trashing your production system. Oct 28, 2010 at 22:44
  • @Evan I'd rather have both, restore from tape can take many hours, but restore from local, or direct-attached disk could be done in minutes. Oct 29, 2010 at 8:53
  • @Tim O'Connor: D2D2T is great when you can get it. Keep in mind that restoring individual items from disk or tape can be very fast. Disk-based backup has a reputation for being fast to restore from, but most people are thinking "access the data directly from the B2D media" not "restore it" when they say that. If you have to restore a couple of TB of data from a disk-based backup system to, say, a replacement SAN after yours burned up in a fire it's not going to be "minutes" to have that data copied over. Disk and high-end tape, in terms of data transfer speed, are very similar. Oct 29, 2010 at 14:13

7 Answers 7

13

There are many ways of handling data that size. A lot of it depends on your environment and how much money you're willing to spend. In general there are a few overall 'get the data off the server' strategies:

  • Over the Ethernet Like it says on the box, data is streamed to Some Where Else for handling. 20TB will take a long time to copy over 1GbE, but it can be done. Hardware can help (such as 10GbE links, or in some cases NIC bonding).
  • Over the Storage subsystem If you're on Fibre Channel, send it to another device on the FC network. If you've got SAS, send it to a SAS-attached device. Generally faster than Ethernet.
  • Send it to another disk array Send it to another hunk of storage attached to the same server.

That's the 100Km view. Once you start zooming in things get a lot more fragmented. As already mentioned, LTO5 is a specific tape technology that designed for these kinds of high-density loads. Another identical storage array is a good target, especially if you can use something like GlusterFS or DRBD to get the data over there. Also, if you need a backup rotation or just the ability to keep running in case the array fails will affect what you put into place.

Once you've settled on a 100Km view method, getting into software will be the next big task. Factors influencing this are what you can install on your storage server in the first place (if its a NetApp, that's one thing, a Linux server with a bunch of storage is another thing entirely, as is a Windows server with a bunch of storage), what hardware you pick (not all FOSS backup packages handle tape-libraries well, for instance), and what kind of backup retention you require.

You really need to figure out what kind of Disaster Recovery you want. Simple live-replication is easier, but doesn't allow you to restore from last-week only just-now. If the ability to restore from last week is important to you, then you need to design for that sort of thing. By law (in the US and else where) some data needs to be preserved for 7+ years.

Simple replication is the easiest to do. This is what DRBD is designed to do. Once the initial copy is done, it just sends changes. Complicating factors here are network locality, if your 2nd array is not near to the primary DRBD may not be feasible. You'll need a 2nd storage server with at least as much storage space as the first.


About tape backup...

LTO5 can hold 1.5TB of data w/o compression. Feeding these monsters requires very fast networking, which is either Fibre Channel or 6Gb SAS. Since you need to back up more than 1.5TB in a whack you need to look into autoloaders (here is an example: link, a 24 slot 1-drive autoloader from HP). With software that supports them, they'll handle changing tapes mid-backup for you. They're great. You'll still have to pull tapes out to send to off-site, but that's a damn sight better than hanging around all night to load tapes yourself when the backup calls for them.

If tape gives you the 'legacy, ew' heebiegeebies, a Virtual Tape Library may be more your speed (such as this one from Quantum: link). These pretend to be tape libraries to backup software while actually storing things to disk with robust (you hope) de-duplication techniques. The fancier ones will even copy virtual-tapes to real-tapes for you, if you like that sort of thing, which can be very handy for off-site rotations.


If you don't want to muck about with even virtual tapes, but still want to do direct-to-disk backups, you'll need a storage array sized big enough to handle that 20TB, plus however much net-change data you want to keep a hold of. Different backup packages handle this differently. Some de-duplication technologies are really nice, others are hacky kludges. I personally don't know the state of FOSS backup software packages in this area (I've heard of Bacula), but they may be sufficient. A lot of commercial backup packages have local agents you install on servers to be backed up in order to increase throughput, which has a lot of merits.

1
  • Thank you for the long and thought-out answer. You've given me a lot to ponder :-p Oct 28, 2010 at 23:11
9

LTO-5 jukebox? you'd need somewhere between three and 15 tapes to back that array up, which isn't a crazily large number. The jukebox will take care of changing the tapes for you, and good backup software (eg bacula) will keep track of which file(s) are on which tape.

You will also want to consider the time required to back up a file system that large, inasmuch as it is very likely the FS will change during that period. For best results, a file system that supports snapshots would be very helpful, so you can take an instantaneous snapshot and perform full or incremental backups against that, instead of against the live filesystem.

6
  • 1
    I'm not familiar with tape systems. I'm guessing there's no way to do incremental backups. Also, wouldn't it take several hours and involve manually changing the tape drives one after another? That wouldn't be ideal because I would only have that kind of time once a month, and we really don't want to have a month's worth of data at risk. Am I missing something, or are these just accepted inconveniences/risks/limitations of tape backup systems? Oct 28, 2010 at 21:07
  • 4
    Modern tape backup systems are highly automated and robotic:)
    – phoebus
    Oct 28, 2010 at 21:55
  • 3
    Yes, tape backups normally allow for incremental backups. A good backup strategy is to do full backups (long, slow, lots of tapes) monthly or bi-annually, and do daily incremental or differential backups in-between.
    – Brent
    Oct 28, 2010 at 22:28
  • Tape robots are reasonably priced and hold many tapes. As far as doing the backups, why wouldn't there be a way to do incrementals? Finally, most people trigger the backup to run during off hours. If you don't have those, that's an important part of the specification. Oct 29, 2010 at 1:41
  • Yeah, we really don't have off hours. We have hours where it would be more acceptable for the system to be unavailable (like 4am Saturday morning), but the affected systems will be used 24/7 by potentially hundreds of users. Oct 29, 2010 at 2:12
5

You should probably be looking at backing up to disk, since tape will take a long time, and being sequential access, restores will take forever.

Definitely take advantage of differential or incremental backups - only backing up changes, at whatever frequency makes sense for you.

Probably the ideal solution would have a 2nd similarly sized server at another location, where incremental backups are sent regularly, and that could be swapped into place quickly if the main server ever died. However another option would be to use removable drives on-location, which are then taken offsite for storage.

When you're dealing with that much data, it also makes sense to break up your backups into smaller backup jobs, and if they can't all be backed up every day, stagger your backups so set A gets backed up one day, and set B the next.

Always be thinking about the restore procedure. We got stung once when we had to restore a file from a several-hundred-gig backup job, which took a-lot of memory and a-lot of time to re-build the backup index and restore. In the end we couldn't complete it in a day, and had to build a dedicated restore server to allow our main backup server to continue it's nightly jobs!

--added--

You also want to be thinking about deduplication technologies, which can save huge amounts of space by not backing up the same information multiple times, for multiple users. Many backup solutions or filesystems offer deduplication as a part of their functionality.

4
  • 1
    +1 for thinking about the restore procedure. Amen! Oct 28, 2010 at 22:42
  • Lots of great tips. Thanks. I have a lot of thinking to do. Oct 28, 2010 at 23:11
  • 2
    I'd like to upvote, but I don't see tape mentioned. Tape is, very likely, going to be a vital part of a backup regime for that amount of data if any significant retention window combined with off-site storage is needed. The cost of LTO-5 cartridges for long-term off-site storage, as compared to removable hard disk drives, makes them very attractive. Tape cartridges are also designed for archival storage whereas removable hard disk drives are, typically, not. Oct 28, 2010 at 23:18
  • @Evan: To be fair, he did mention tapes in the very first sentence. Oct 30, 2010 at 4:09
2

First, enumerate the risks you are protecting against. Some common risks:

  • Disaster: Something very unfortunate happens to your entire site.
  • Human errors (this is the one that happens _all_the_time_):
    • Someone decides to exercise the "hot-swap" capability of your storage server in a way not intended by the manufacturer.
    • Someone runs a process that silently corrupts data, which is backed up reliably for a couple of months before the problem is noticed.
    • Someone deletes the important report that is due in an hour and is worth thousands of dollars.

Then evaluate the cost of the various risk avoidance solutions, e.g.:

  • Off-site, on-line backup (remote mirror): Safe from disaster, some (but not all) human error (it is still on-line).
  • Off-site off-line storage (tapes): Safe from disaster, hard to recover data quickly.
  • On-site on-line backup (mirror): Safe from some human error, some hardware failure, vulnerable to disaster.
  • On-site off-line backup (tapes in tape changer): Safe from most human error, most hardware failure.

Then evaluate rotation strategies (how far back do you want to be able to recover, how much data can you afford to lose).

Then pick what your data is worth.

2
  • Nice break down. I've already evaluated this for the most part and landed on the Off-site, online backup option. The purpose of the backup is mostly to protect from disaster in addition to the obvious human error. The rack is located within 2 miles of the gulf coast, so hurricanes are a concern. We'll just have to do our best to protect against the human errors with frequent integrity checks. Your answer helped me feel better about this conclusion. Thanks. Oct 29, 2010 at 2:09
  • I'm glad I could help. Some comments regarding your chosen solution: This may go without saying, but the backup site should probably be in another state or in a place well protected from tbe hurricanes you are subject to. You can mitigate corruption concerns by having a long 'tail' (backups from a wide range of dates in the past). With an online backup, you also want to consider the danger of accidentally deleting the data instead of restoring it. Finally, always test your restore process. Oct 30, 2010 at 5:32
2

I have a customer with two similar 12 TB systems in two different buildings, connected at 1GB. One is the production system; it's backed up incrementally (with daily snapshots) to the other with the great rdiff-backup utility. rdiff-backup must be available in your standard distribution repository.

1

Off-site, on-line backup (remote mirror)

use rsync though ssh (only changes) - first backup has to be done locally, but after that backup will be a breeze depending on changes

if you need to keep versions with changes- rdiff-backup

http://www.nongnu.org/rdiff-backup/

btrfs file system in Linux sounds promising, but still on heavy development

1
  • Thanks for pointing me toward rdiff. I already use rsync, and this looks like the perfect step up from that. Oct 31, 2010 at 2:19
1

Take a look at your actual "content" and how often it changes before you plan your strategy. Many times people just churn the same data to tape weekly over and over for no good reason.

Deduplication technologies from some vendors can allow snapshotting to save you from individual file restores but you will always need offsite for protection.

4
  • The system will be used by thousands possibly tens of thousands of daily users entering forms and updating information. This is highly dynamic data. I should have mentioned that in the question. Oct 30, 2010 at 0:41
  • If it were me, I would design the system with enough overhead or snapshot capability that I wouldn't have to go to the real backups unless it's a disaster. Oct 30, 2010 at 15:09
  • I agree. As I said before, the drives will be in RAID 10, so we're covered in case of hard drive failure, and I will have local backups/snapshots as well. The offsite backup is for the worst-case scenario like a meteor hitting the co-locate or someone accidentally running rm -rf /* on the storage server. Oct 30, 2010 at 16:19
  • Well, I was referring to overhead in regards to capacity. RAID10 is smart for best redundancy of course, but I'd take RAID6 if performance wasn't as much of a requirement and if I could use the extra space for more snapshot area. The more snapshot you can afford, the less you'll need "backup" for file restores. Oct 31, 2010 at 23:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .