Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have about 15TB of data across several RAID6 arrays, and it's been growing at a rate of about 10%. I want a cheap solution to back up this data at regular intervals and bring offsite. So I purchased five 3TB drives, hoping to just back them up using a eSATA drive dock. Ideally, I wanted to load up 3TBs at a time, and keep popping in drives until I was done.

However, the filesystems I'm backing up have deep and complex folder structures, so that would rule out doing a folder by folder backup especially as I plan on doing this every few weeks. Incremental backups would be ideal, and I just keep adding drives as needed. However, I wasn't able to find an affordable solution to this (Roxio Retrospect is too pricey).

Does anybody have any suggestions?

Thanks

share|improve this question

migrated from superuser.com May 16 '11 at 13:41

This question came from our site for computer enthusiasts and power users.

7  
"It's been growing at a rate of about 10%" --> is that per month, per hour, per decade...? –  Mehrdad May 15 '11 at 5:55

3 Answers 3

Assuming you're using linux, you could probably create a single drive volume with greyhole then load them up, and let it take care of the complexity. However this wouldn't work unless you could load up the whole set of drives at once.

There's also applications that split folders of files files into known sizes - dirsplit comes to mind. I have no idea how they'd work with multiple swappable drives tho.

share|improve this answer

Don't try to roll your own backups - there are great software packages out there that have already solved this problem and solved it well. I'm a big fan of Amanda and Bacula.

You should be looking into a proper backup solution; if you can buy a tape drive or library, do so if you actually care about the data you're backing up.

If you get a single drive, go with Amanda. If you get a library, go with Bacula.

If you insist on sticking with your eSATA hard drive backup process, I'd suggest using Amanda and using the hard drives as 'tapes' from Amanda's point of view.

share|improve this answer
    
Agreed. You need some way for the backup software to keep track of where the data is on the drives in case you need to do a restore. The reason why is because you're spanning the backup data across multiple drives, & as you mentioned, doing this manually would be too difficult. –  churnd May 16 '11 at 15:00

With that much data, you should really look at Nexenta, The URL is for the free community edition that will handle up to 18 TB but they do commercial ones that handle much, much more. The nice thing about Nexenta is that you do not need RAID controllers, therefore they are cheaper, and you can use any collection of drives, i.e. they don't need to be a matched set.

But the main benefit of Nexenta is the deduplication and compression in the ZFS filesystem. Depending on what you are storing, it may take quit a bit less raw drive space, which also saves you money. Then you either take the whole system offsite, or you build it with external SATA cables in the first place and just unplug the drives. ZFS knows which drive is which so you do not need to keep track of the cables that they were plugged into. No need to buy a case for all the drives, just make up a nice plastic storage box with some conductive foam dividers for moving the drives.

And if you are not sure how this will all work, get a couple of USB hubs, and a dozen thumbdrives, and set up Nexenta with that. Then you can see in practice how the ZFS RAID works, and how it recognizes the drives by content, not by connection.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.