Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are moving ~60 user mailboxes from a Postfix/Dovecot/Maildir setup into Exchange 2007. I am aware of Microsoft's Transporter Suite, which will allow for an IMAP-to-Exchange conversion of mailboxes in bulk. However, there is a concern about how to clean up a botched process, should there be an unforeseen issue. Apparently, attempting to remove a corrupted Exchange mailbox for a user also results in a rather scary-sounding dialog that implies the user account will be removed as well, not just the corrupted box itself. While the original email store (in Maildir format) is available, we would need the ability to remove the bad box(es) in question and repeat the process, so this method is not a viable option.

I have grave concerns at this point about affecting a batch- or bulk-style transition without massive manual intervention. We are trying to avoid visiting 50+ desktops and manually moving email by using Outlook or Thunderbird as a "pivot" agent; while this is 100% guaranteed to work, it will take weeks or months to move over 2.5 million messages one account at a time.

So I have begun to look into other tools and methods, both open-source and commercial.

The first one I tried was OfflineIMAP, which as it turns out, is not very Exchange-friendly; a subtle effect of the migration process results in exhaustion of Exchange's named properties due to a unique header being generated for each email moved. A patch is available for this that changes the unique header into a single, generic header that has a unique identifier after it, avoiding the exhaustion issue. Yet, after applying the patch, there are still some issues. A pity, really, as it would have been nice to keep them in sync.

On the commercial side, I took a brief look at Transend Migrator in a trial mode. The results were mixed and there appears to be little documentation delivered with the binary. This is hardly encouraging, especially for something that end-users will readily notice if it goes horribly wrong.

There are other issues as well. The mailboxes are stored as Maildir, but the box names conform to Thunderbird's expectations (we've been on TBird for several years). Many box names do not match (Sent vs. Sent Items, Trash vs. Deleted Items, etc.) and will need to be translated effectively as they are ported. Yes, we pamper our end-users that much - a successful migration will consist of informing them we are switching to Outlook and giving everyone a small 30 minute class, followed by people openning their new Exchange-based mailboxes for the first time.

Someone, somewhere surely has something that works...

EDIT: A follow-up

The larch script, written in Ruby, has provided the easiest solution, although with a few caveats. Here's a heavily condensed version of what happens:

  1. Obtain the user's username and password. Have the password reset after the transition.
  2. Connect to the user's box via IMAP and ensure that the INBOX is subscribed, and that all messages are marked as read. There is an issue on the Exchange side that can prevent messages from successfully importing if the message is unread and in a certain state...
  3. Change the postfix transport map to point to the exchange server, which causes all new deliveries to arrive at the Exchange server instead of the current email server.
  4. Use the larch command to move the email over, omitting the user's Trash can. We had to capitulate and omit this because there were users turning the Trash into a personal filing system.
  5. Note any messages that didn't transition. Re-examine and attempt to recopy them again. This will only move the newer messages.
  6. Create a record in MySQL that instructs Dovecot to go into proxy mode for that user, and point the record at the Exchange server. This allows existing clients to connect unchanged until we are ready to set up their Outlook install.
  7. Re-connect to the user's Inbox and check to see that all messages are copied.
share|improve this question
    
An update: I am closing in on a potential solution. I will post the result here as a follow-up for those who want to know, but I refuse to answer my own question - instead, I'll award the answer to the person whos suggestion is closest in practice. Thank you for all of your replies, I am looking diligently into each. –  Avery Payne May 19 '09 at 0:34
    
Last update: at this point, it looks like the larch script, written in Ruby, will provide us with an adequate tool. I have upvoted everyone at least once because all of them were excellent suggestions, even though they didn't pan out in my situation. –  Avery Payne May 24 '09 at 5:40
add comment

5 Answers

up vote 1 down vote accepted

The best way to go is using imap!

You COULD let each user move the messages himself by giving him both accounts in Outlook.

I did a migration from Cyrus to Exchange 2003 using a separate linux box with the Debian package "uw-mailutils" and this command:

mailutil transfer -verbose {imap.xxx.org:143/imap/user=m_xxx}INBOX {exchange.xxxx.org:143/imap/user=m_xxx@exchange.xxxx.org} INBOX

mailutil is from the UW-IMAP project. Every Linux distro should contain it.

When it failed (which it didn't) I would just go into the exchange account and delete "INBOX"

share|improve this answer
    
This is a nice try, but it won't work for a few reasons. The first is, as I mentioned, we really can't delete the box on '07. The second, as I also mentioned, is that we don't want end-users doing this by hand, it will take forever (400 messages is a 5-10 minute proposition right now). I haven't even mentioned how Exchange does (not) handle email with corrupted attachments...still, nice try. –  Avery Payne May 15 '09 at 0:00
    
(wish the comments would be expanded) I will give mailutil a try tomorrow some time. –  Avery Payne May 15 '09 at 0:03
    
Well I did not have to start over with any user but the first when I did it. mailutil does not create new headers. You can use multiple ssh-sessions (use screen!) to increase the performance maybe. Your folder names should not be an issue, mailutil will read the maildir (over imap...) and Exchange can surely allow any folder name and kind of hierachie (e.g. mails inside folder vs. folder with mail an folder with folders as with mbox). It worked for me for 30 users and went fast. We did it user by user, not over the weekend. mailutil transfer will not delete the old mailbox,so give it a try! –  Christian May 15 '09 at 0:20
    
I ended up with a different utility, although the results were acceptable. You received the answer vote because your description was closest to the approach I took. For the others, I do thank you for your input, and it was valuable; you have been upvoted accordingly. –  Avery Payne May 24 '09 at 5:41
add comment

I've done many such migrations on much larger systems; 300+ mailboxes and I've migrated in many directions: Courier -> Cyrus, Exchange -> Cyrus, Courier -> Exchange, and Cyrus -> Exchange...

Our tool of choice is called imapsync; an open source Perl script. It uses the actual IMAP protocol to handle migrations thus alleviating the need to deal with underlying specifics to each implementation (aildir formats, mailbox annotations such as DONT (.) vs. slash (/) seperators.

IMAPSYNC is also re-entrant safe. You can run it multiple times over the same accounts and it will only copy what has not been copied the first time through or any new emails that may have arrived.

We typically dump all the user accounts to a BASH script that, in the end, is executed like so:

./imapsync --host1 mail.pozicom.net --user1 someone@domain --password1 mypassword \
           --host2 localhost --user2 someone@domain --password2 mypassword

Where host1 is where you're copying from host2 is where you're copying to.

You can get IMAPSYNC from here: http://freshmeat.net/projects/imapsync/

share|improve this answer
add comment

I've used the "imapsync" program before to mmigrate from an IMAP server to Scalix (an Exchange-like system). Think rsync for IMAP mailboxes.

Jim.

share|improve this answer
add comment

We have a huge large mail system, and routinely ingest users from other mailsystems (typically Notes or Groupwise). One common thread in all of these migrations is that the data migration piece is always a disaster. There are always intractable problems, VIP issues end up taking priority and the normal user population suffers. Migration tools have bugs that never get fixed because nobody does a Notes->Exchange migration twice!

What we're looking at for a future strategy is a mostly greenfield mailbox, with users identifying email that must be retained. For you its easy -- keep the dovecot servers running for 6 months, setup Outlook profiles pointing to the old imap system (I believe this is scriptable with Outlook 2007) and let users migrate in their own time.

You'll find that most folks really don't need what they say they need, and after 6 months, any legit need to move data to the new system will be done.

Another option is to use a mail archiving solution. We have to archive all email due to litigation, and oftentimes you can "ingest" messages from an old system into a hosted archiving solution. Check out Postini and Microsoft Hosted services.

share|improve this answer
add comment

Use the Transporter Suite. It was written specifically for this purpose.

Basically, you import a CSV file of the IMAP users and passwords, and it will import the data correctly. You can also import users one at a time if you are scared of doing a batch process. It's actually pretty hard to botch an import.

Do not use imapsync or any other IMAP to IMAP based sync engine. It sort of works, but much of the metadata will be screwed up -- specifically dates. Even with the --syncinternaldates option, Exchange will still overwrite the dates to the current date, and you will have a botched migration.

share|improve this answer
    
We ended up with an IMAP-based solution anyways. The Transporter Suite was not an option because of concerns on "reversability". And the dates on messages came out just fine. –  Avery Payne Jul 30 '09 at 0:20
    
Glad they fixed the date bug then. What were the concerns on "reversability?" –  Jesse Weigert Aug 2 '09 at 6:10
    
@Jesse Weigert, our Windows admin pointed out that once you had a box on the Exchange server, the current (2007) implementation wants to delete the user account as well as the user's box, which is a big problem. You can't just "delete the box" and start over when that user is actively using the account on the network. –  Avery Payne Sep 16 '09 at 17:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.