Backup solution for 10TB using ubuntu

Question

I am setting up a backup server with roughly ten 2TB hard drives raided. The purpose of this machine is simply to back up about 3-10TB of data from another server.

What is the best way to achieve this backup. It would be nice to have version history. I thought about just setting up gitosis and having a git repo. The other machine will just git commit and git push onto this server at certain intervals. But I'm not sure git can handle TBs of data like this. The files consist of 90% images (jpeg, tiff, etc that wont change, so these are small files), and 10% are big database dumps and will change daily.

Would the best solution be to rsync it to the backup machine and use LVM to take snapshots? How about using TimeVault? I would like to have not just one copy of the backup, but multiple versions at different time intervals. Any information regarding this issue would be great.

Answer 1

I'd setup the box as a NAS with NextentaStor Community Edition (ZFS!) or OpenFiler.

Why bother with a full distro unless you have plans to use it as something else? Less things to go wrong because it's purpose-built with a smaller footprint; both OpenFiler and NextentaStor have their pros and cons, but either would be a better option for a pure storage appliance than straight-up Ubuntu.

Answer 2

Using git doesn't seem like the right fit. If you really prefer it that way, have a look at git bup which is a git extension to smartly store large binaries in a git repo.

That said, I recommend rsnapshot, rdiff-backup.

I most certainly DO NOT recommend LVM snapshots for this1.

• The write performance will degrade badly
• On those volumes, a snapshot will cause boot times in the many minutes, if not hours (here)
• there are fatal gotchas when running out of space
• and the last time I checked anything like a rollback was still a remote promise
• Mind you that even mounting a snapshot alongside your live filesystem could prove to be very tricky because of filesystems relying on guids to be unique in the fs header
• Also, barring the use of iSCSI or DBRD (etc) you're stuck on the same host as the live fs, making the backup far less useful (and degrading performance even more)

For this kind of scenario, I prefer ZFS (send, receive). To be honest, I think zfs-fuse might be too slow (but test it!) at the moment, but zfsonlinux seems to come along nicely and might give you a lot to work with.

---

1 I just retrieved this tidbit I wrote earlier about this subject:

_{However, I can no longer count the different failure modes I encountered when using snapshots. I've stopped using them altogether - it's just to dangerous.}

The only exception I'll make now is my own personal mailserver/webserver backup, where I'll do overnight backups using an ephemeral snapshot, that is always equal the size of the source fs, and gets deleted right afterwards.

Most important aspects to keep in mind:

1. if you have a big(ish) fs that has a snapshot, write performance is horribly degraded
2. if you have a big(ish) fs that has a snapshot, boot time will be delayed with literally tens of minutes while the disk will be churning and churning during import of the volume group. No messages will be displayed. This effect is especially horrid if root is on lvm2.
3. if you have a snapshot it is very easy to run out of space. Once you run out of space, the snapshot is corrupt and cannot be repaired.
4. Snapshots cannot be rolledback/merged at the moment (see http://kerneltrap.org/Linux/LVM_Snapshot_Merging). This means the only way to restore data from a snapshot is to actually copy (rsync?) it over. DANGER DANGER: you do not want to do this if the snapshot capacity is not at least the size of the source fs; If you don't you'll soon hit the brick wall and end up with both the source fs and the snapshot corrupted. (I've been there!)

Answer 3

I'd highly suggest not using git for this. Sure it may work, but it's pretty sub-optimal.

You could you rsync, LVM, and snapshots if you'd like. My preferred backup method for instances like this is to use rnapshot or rdiff-backup. They can leverage the optimizations that rsync gives you, while providing an incremental set of backups at the same time.

Answer 4

Rsync's "--backup-dir=" option could eliminate the need for daily snapshots. Any files that get changed are dropped into a backup folder and can be restored from there.

I have a nightly script pulling data from remote sites into a central backup, the backup directory is scripted to be whatever date the backup was done so I have a version history of every file that's changed going back as long as my storage will hold.

If you want I could post the script, it's pretty ugly and specific to our needs tho so I wouldn't expect it to be of much use.

Answer 5

I've set up a smaller backup server using BackupPC. It's in the Ubuntu repositories, setting it up is a snap. Uses rsync for transfer, does file-level deduplication.

It will keep version history, and you can specify how many to keep going in the past. As they get older, it auto removes some of them. The assumption is that the further in the past that you go, the lest granularity you will need. It can be adjusted to whatever you want, though.

Check it out, it's really good.

https://help.ubuntu.com/community/BackupPC

http://backuppc.sourceforge.net/faq/BackupPC.html