0

I'm facing a rebuilding of the volume on which I host the mail storage (kerio mailserver, which uses maildirs). I need to backup and restore as quick as possible the 3.5+ millions (for about 600GB) small files of the store directory. It takes more than 12 hours via rsync to a NFS share, but I also have a 1TB firewire 800 raid1 disk that I can use (from some preliminary tests it's faster). I'm working off a XServe intel. What is the fastest way to do it? Rsync? Finder copy? tar?

3 Answers 3

3

I consider rsync to be one of the best options in this case. You may want to skip the Mac OS X built in rsync 2.6.9 and go for rsync 3 which is usually way faster. You can get a precompiled Mac optimized universal binary of rsync 3.0.6 with mlbackup, a free rsync based backup utility. (Installs rsync in /usr/local/maclemon/bin/rsync) For full and fair disclosure: I am the author of mlbackup.

If this server is your live mailserver, you may want to do an rsync copy to your FireWire device and jut let it take the time it needs to (and take time it will). When you're ready to go you just rerun the rsync command to have it updated to the current status, which will take significantly less time.

I'd suggest you just rsnyc to another drive, and run the mailserver from the new drive. That way you only need to copy once. Don't know if you have a spare bay left in your Xserve, or if you can run from the external drive for a few days. With this technique you can slit the backup and restore-to-internal-drive processes into to distinct events which you could probably do on two weekends to keep service downtime at a minimum.

Just make sure you have more than one backup of the data, just in case. You don't want to have anything go wrong.

You can also have a look at Alex's excellent mailbfr which will work up until 10.5.8, a Snow Leopard compatible version is in the works and should be released in the not too distant future.

2

600GB over 12 hours is 50GB/hr or 111Mb/s as a reference so assuming Gigabit Ethernet that isn't that quick. The firewire drive will probably be a faster option but do you trust it as your only backup of this data over the NFS share? In fact do you really want only one backup of this data?

Have you tried splitting the backup into multiple rsync processes as per Parallelizing RSYNC Processes? This will put more load onto your CPU and IO so you need to test how many simultaneous rsync processes you can run at one time.

2
  • 1
    From my experience rsync is very likely to max out the IO performance on the Xserve, even with internal drives, and especially with FireWire devices.
    – MacLemon
    Mar 28, 2010 at 12:53
  • Thank you for the tip! Indeed the bottleneck here is the indexing. Since my structure is mailserver/store/domain1/user1, mailserver/store/domain2/user2 and so on, I can get my awk skills up to speed and write a script to parallelize rsync. Oh and obviously I will first backup to NFS the (live) server, when that is done I'll stop it and backup to firewire, so I have the data in two places, one slightly outdated. Mar 29, 2010 at 12:07
1

The bottleneck, I suspect, is the incremental write to disk on the NFS server. If you have the capacity, to do it, I suspect that you'll do better if you either stream it through tar, or tarball it up and then move it over directly. You'll loose the incremental aspect of the rsync, but I think there's probably more overhead in the rsync-related file system crawling than there is in the actual copying. Particularly if the distance to the NFS server isn't that far and there's a lot of bandwidth there (as it sounds like there is.)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .