Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So here's the situation.

I've got about 20,000 maildir email accounts chewing up a several hundred GB of space on our email server. Maildir by nature keeps thousands of tiny a** little files, instead of one .mbox file or the like...

So i need to migrate all of these several millions of files from one server to the other, for both space and life-cycle reasons.

the conventional methods i would use all work just fine. rsync is the option that comes immediately to mind, however i wanted to see if there are any other "better" options out there.

Rsync not handling multi-threaded transfers in this situation sucks because it never actually gets up to speed and saturates my network connection, because of this the transfer from one server to another will take hours beyond hours, when it shouldn't really take more then one or two.

I know this is highly opinionated and subjective and will therefore be marked community wiki.

share|improve this question
    
Have you tried using rsync in daemon mode, rather than the default transport method (tunnelling over ssh)? –  Steven Monday Jan 9 '11 at 16:31
add comment

2 Answers

up vote 3 down vote accepted

consider a tar pipe like this one :

tar cf - . | ssh remote "cd /backup; tar xf -"

I would also be looking at bbcp. I realise the page about bbcp is enormous and confusing, but take the time to read it because I feel its probably the best solution here.

http://www.slac.stanford.edu/~abh/bbcp/

I've done bbcp transfers on a variety of networks and found with the right thread settings you can normally saturate any network.

Think carefully about the filesystem you are using as ext3 is likely to be very inefficient for lots of small files. I would be using XFS.

share|improve this answer
    
I second the idea of using a different destination filesystem like xfs, although I think ext3 does have better lots of small file handling than ext3 too. –  Phil Hollenback Jan 9 '11 at 19:08
1  
I've actually already discussed this and XFS will be our new File system. Thanks for seconding (and thirding) my ideas on this matter. –  GruffTech Jan 10 '11 at 19:15
add comment

Depending on your service requirements, you might also want to consider using a combination of nginx's IMAP proxying and imapsync.

Proxy the IMAP requests dynamically (using the authentication lookup to determine which backend server the user should connect to). Iterate over your user list with imapsync. Once the user has migrated to the new server, lock the account, run a final sync, and update a record somewhere so the auth lookup returns the new server.

This way you generate no visible outage to the end-user.

imapsync is also quite clever in how it does incremental syncs.

I have done this to migrate users from Courier to Cyrus, and between mailstore backends.

share|improve this answer
    
this is strictly from a file standpoint -- using imap as a middle man would be a horrifically inefficient way of doing it in my opinion. –  GruffTech Jan 10 '11 at 19:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.