Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm facing a rebuilding of the volume on which I host the mail storage (kerio mailserver, which uses maildirs). I need to backup and restore as quick as possible the 3.5+ millions (for about 600GB) small files of the store directory. It takes more than 12 hours via rsync to a NFS share, but I also have a 1TB firewire 800 raid1 disk that I can use (from some preliminary tests it's faster). I'm working off a XServe intel. What is the fastest way to do it? Rsync? Finder copy? tar?

share|improve this question
add comment

3 Answers

I consider rsync to be one of the best options in this case. You may want to skip the Mac OS X built in rsync 2.6.9 and go for rsync 3 which is usually way faster. You can get a precompiled Mac optimized universal binary of rsync 3.0.6 with mlbackup, a free rsync based backup utility. (Installs rsync in /usr/local/maclemon/bin/rsync) For full and fair disclosure: I am the author of mlbackup.

If this server is your live mailserver, you may want to do an rsync copy to your FireWire device and jut let it take the time it needs to (and take time it will). When you're ready to go you just rerun the rsync command to have it updated to the current status, which will take significantly less time.

I'd suggest you just rsnyc to another drive, and run the mailserver from the new drive. That way you only need to copy once. Don't know if you have a spare bay left in your Xserve, or if you can run from the external drive for a few days. With this technique you can slit the backup and restore-to-internal-drive processes into to distinct events which you could probably do on two weekends to keep service downtime at a minimum.

Just make sure you have more than one backup of the data, just in case. You don't want to have anything go wrong.

You can also have a look at Alex's excellent mailbfr which will work up until 10.5.8, a Snow Leopard compatible version is in the works and should be released in the not too distant future.

share|improve this answer
add comment

600GB over 12 hours is 50GB/hr or 111Mb/s as a reference so assuming Gigabit Ethernet that isn't that quick. The firewire drive will probably be a faster option but do you trust it as your only backup of this data over the NFS share? In fact do you really want only one backup of this data?

Have you tried splitting the backup into multiple rsync processes as per Parallelizing RSYNC Processes? This will put more load onto your CPU and IO so you need to test how many simultaneous rsync processes you can run at one time.

share|improve this answer
1  
From my experience rsync is very likely to max out the IO performance on the Xserve, even with internal drives, and especially with FireWire devices. –  MacLemon Mar 28 '10 at 12:53
    
Thank you for the tip! Indeed the bottleneck here is the indexing. Since my structure is mailserver/store/domain1/user1, mailserver/store/domain2/user2 and so on, I can get my awk skills up to speed and write a script to parallelize rsync. Oh and obviously I will first backup to NFS the (live) server, when that is done I'll stop it and backup to firewire, so I have the data in two places, one slightly outdated. –  ams0 Mar 29 '10 at 12:07
add comment

The bottleneck, I suspect, is the incremental write to disk on the NFS server. If you have the capacity, to do it, I suspect that you'll do better if you either stream it through tar, or tarball it up and then move it over directly. You'll loose the incremental aspect of the rsync, but I think there's probably more overhead in the rsync-related file system crawling than there is in the actual copying. Particularly if the distance to the NFS server isn't that far and there's a lot of bandwidth there (as it sounds like there is.)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.