1

I'm currently using the traditional rsync+cp -al method to create incremental/snapshot backups of our server tree. The backups are going onto a pair of eight-disk towers connected to the backup machine (a Sandy Bridge machine with 16 GB of RAM, running CentOS 5.5) via four eSATA connections (four disks per connection). Each disk is a regular 2 TB disk, so we have 32 TB of disk space connected to the backup machine. We're backing up about 20 TB of data on the servers with this.

The problem is that each daily backup is taking more than 24 hours, and the real time-killer isn't the actual rsync, but the time it takes to perform a cp -al of the tree locally on the backup machine. It's taking more than 12 hours just to make the shadow copy of the tree, and as far as I can tell the performance backlog is at the disk (top shows the cp using a lot of RAM but not a lot of CPU and mostly in uninterruptible-sleep state)

We have the server data split into four major volumes (and a few minor ones), and each of these backups runs in parallel (with some offsets in the cron to try to get some disks' cp done first). There are two volumes on the backup drive, both striped LVM volumes of 16 TB each.

So obviously I need to improve the performance because it's unusable as it stands.

The first question is: when CentOS 6 comes out, with support for btrfs, will making snapshots of subvolumes with btrfs substantially increase this performance?

The second is: is there a way, with ext3 or something else supported in CentOS 5 or 6, to 'encourage' it to put the directories/inodes in one part of a volume (which could happen to be the part that's on an SSD, via LVM) and the files in another? That would presumably solve the problem, but I don't know of ways to hint ext3 like that.

Improve this question
1
  • SSD won't help you even if there was a way to do what you ask. LVM doesn't support TRIM on CentOS, so your writes are probably going to be significantly slower than to a spinning disk.
    – jgoldschrafe
    Feb 10, 2011 at 20:07

2 Answers 2

Reset to default
2

You could try using --link-dest and related options instead of following rsync with cp -al. See the man page and tutorials like this one for details.

You could get a little extra speed by turning off ext3's journal, though I wouldn't recommend that as it increases the chance of damaged backups if something goes wrong during an update.

If your backups include directories with many files, then you might find reformatting as ext4 or using the dir_index option with ext3 may improve things - but in both cases you might need to reformat to see the full benefit as just remounting the filesystem with the new options won't convert any existing structures.

Improve this answer
3
  • does updating the access time cause a file to "change" and therefore be backed up in the next incremental?
    – SpacemanSpiff
    Feb 10, 2011 at 20:52
  • Somehow or other --link-dest has way better performance than cp -al. That doesn't seem logical to me, but my backup time is down from over 24 hours to about 6. Many thanks!
    – Alistair Bell
    Feb 11, 2011 at 16:00
  • A large part of the improvement over a very large set of files and directories will be due to the main image and the snapshot being updated at the same time, so the relevant part of the directory structures will be in the directory cache when needed far more often. When running cp -al after the main sync operation this boost from cache hits is usually lost. See linux-tutorial.info/modules.php?name=MContent&pageid=285
    – David Spillett
    Feb 12, 2011 at 12:03
0

Consider using rdiff-backup instead of rsync+cp. It handles old copies of files automatically, so you don't need to cp.

From rdiff-backup page:

"The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago."

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .