6

I have about 15TB of data across several RAID6 arrays, and it's been growing at a rate of about 10%. I want a cheap solution to back up this data at regular intervals and bring offsite. So I purchased five 3TB drives, hoping to just back them up using a eSATA drive dock. Ideally, I wanted to load up 3TBs at a time, and keep popping in drives until I was done.

However, the filesystems I'm backing up have deep and complex folder structures, so that would rule out doing a folder by folder backup especially as I plan on doing this every few weeks. Incremental backups would be ideal, and I just keep adding drives as needed. However, I wasn't able to find an affordable solution to this (Roxio Retrospect is too pricey).

Does anybody have any suggestions?

Thanks

1
  • 9
    "It's been growing at a rate of about 10%" --> is that per month, per hour, per decade...?
    – user541686
    May 15, 2011 at 5:55

4 Answers 4

3

Don't try to roll your own backups - there are great software packages out there that have already solved this problem and solved it well. I'm a big fan of Amanda and Bacula.

You should be looking into a proper backup solution; if you can buy a tape drive or library, do so if you actually care about the data you're backing up.

If you get a single drive, go with Amanda. If you get a library, go with Bacula.

If you insist on sticking with your eSATA hard drive backup process, I'd suggest using Amanda and using the hard drives as 'tapes' from Amanda's point of view.

1
  • Agreed. You need some way for the backup software to keep track of where the data is on the drives in case you need to do a restore. The reason why is because you're spanning the backup data across multiple drives, & as you mentioned, doing this manually would be too difficult.
    – churnd
    May 16, 2011 at 15:00
2

Assuming you're using linux, you could probably create a single drive volume with greyhole then load them up, and let it take care of the complexity. However this wouldn't work unless you could load up the whole set of drives at once.

There's also applications that split folders of files files into known sizes - dirsplit comes to mind. I have no idea how they'd work with multiple swappable drives tho.

2

I know this is an old question. Just wirting this so people who searches for a solution can see it.

If you are using Linux, you can simply use tar. It offers option M which allows you to create a multi-volume archive. Assume your esata dock shows up as /dev/sdX in your linux system.

tar Mcvf /dev/sdX /your_folder

When the first disk is full, it will prompt you to swap the disk, once the replacement disk is ready, press Enter to continue.

To extract the files, load the first backup disk and use

tar xvf /dev/sdX -C /your_restore_dir/

It will prompt you to swap the disk similar to the back up process.

Note that this uses the entire target disk as a block device. It won't create any partition nor file system on the target disk, so don't partition or format the backed up disks.

Reference: https://www.gnu.org/software/tar/manual/html_node/Multi_002dVolume-Archives.html

1

With that much data, you should really look at Nexenta, The URL is for the free community edition that will handle up to 18 TB but they do commercial ones that handle much, much more. The nice thing about Nexenta is that you do not need RAID controllers, therefore they are cheaper, and you can use any collection of drives, i.e. they don't need to be a matched set.

But the main benefit of Nexenta is the deduplication and compression in the ZFS filesystem. Depending on what you are storing, it may take quit a bit less raw drive space, which also saves you money. Then you either take the whole system offsite, or you build it with external SATA cables in the first place and just unplug the drives. ZFS knows which drive is which so you do not need to keep track of the cables that they were plugged into. No need to buy a case for all the drives, just make up a nice plastic storage box with some conductive foam dividers for moving the drives.

And if you are not sure how this will all work, get a couple of USB hubs, and a dozen thumbdrives, and set up Nexenta with that. Then you can see in practice how the ZFS RAID works, and how it recognizes the drives by content, not by connection.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .