2

We are moving ~60 user mailboxes from a Postfix/Dovecot/Maildir setup into Exchange 2007. I am aware of Microsoft's Transporter Suite, which will allow for an IMAP-to-Exchange conversion of mailboxes in bulk. However, there is a concern about how to clean up a botched process, should there be an unforeseen issue. Apparently, attempting to remove a corrupted Exchange mailbox for a user also results in a rather scary-sounding dialog that implies the user account will be removed as well, not just the corrupted box itself. While the original email store (in Maildir format) is available, we would need the ability to remove the bad box(es) in question and repeat the process, so this method is not a viable option.

I have grave concerns at this point about affecting a batch- or bulk-style transition without massive manual intervention. We are trying to avoid visiting 50+ desktops and manually moving email by using Outlook or Thunderbird as a "pivot" agent; while this is 100% guaranteed to work, it will take weeks or months to move over 2.5 million messages one account at a time.

So I have begun to look into other tools and methods, both open-source and commercial.

The first one I tried was OfflineIMAP, which as it turns out, is not very Exchange-friendly; a subtle effect of the migration process results in exhaustion of Exchange's named properties due to a unique header being generated for each email moved. A patch is available for this that changes the unique header into a single, generic header that has a unique identifier after it, avoiding the exhaustion issue. Yet, after applying the patch, there are still some issues. A pity, really, as it would have been nice to keep them in sync.

On the commercial side, I took a brief look at Transend Migrator in a trial mode. The results were mixed and there appears to be little documentation delivered with the binary. This is hardly encouraging, especially for something that end-users will readily notice if it goes horribly wrong.

There are other issues as well. The mailboxes are stored as Maildir, but the box names conform to Thunderbird's expectations (we've been on TBird for several years). Many box names do not match (Sent vs. Sent Items, Trash vs. Deleted Items, etc.) and will need to be translated effectively as they are ported. Yes, we pamper our end-users that much - a successful migration will consist of informing them we are switching to Outlook and giving everyone a small 30 minute class, followed by people openning their new Exchange-based mailboxes for the first time.

Someone, somewhere surely has something that works...

EDIT: A follow-up

The larch script, written in Ruby, has provided the easiest solution, although with a few caveats. Here's a heavily condensed version of what happens:

  1. Obtain the user's username and password. Have the password reset after the transition.
  2. Connect to the user's box via IMAP and ensure that the INBOX is subscribed, and that all messages are marked as read. There is an issue on the Exchange side that can prevent messages from successfully importing if the message is unread and in a certain state...
  3. Change the postfix transport map to point to the exchange server, which causes all new deliveries to arrive at the Exchange server instead of the current email server.
  4. Use the larch command to move the email over, omitting the user's Trash can. We had to capitulate and omit this because there were users turning the Trash into a personal filing system.
  5. Note any messages that didn't transition. Re-examine and attempt to recopy them again. This will only move the newer messages.
  6. Create a record in MySQL that instructs Dovecot to go into proxy mode for that user, and point the record at the Exchange server. This allows existing clients to connect unchanged until we are ready to set up their Outlook install.
  7. Re-connect to the user's Inbox and check to see that all messages are copied.
2
  • An update: I am closing in on a potential solution. I will post the result here as a follow-up for those who want to know, but I refuse to answer my own question - instead, I'll award the answer to the person whos suggestion is closest in practice. Thank you for all of your replies, I am looking diligently into each. May 19, 2009 at 0:34
  • Last update: at this point, it looks like the larch script, written in Ruby, will provide us with an adequate tool. I have upvoted everyone at least once because all of them were excellent suggestions, even though they didn't pan out in my situation. May 24, 2009 at 5:40

5 Answers 5

1

The best way to go is using imap!

You COULD let each user move the messages himself by giving him both accounts in Outlook.

I did a migration from Cyrus to Exchange 2003 using a separate linux box with the Debian package "uw-mailutils" and this command:

mailutil transfer -verbose {imap.xxx.org:143/imap/user=m_xxx}INBOX {exchange.xxxx.org:143/imap/[email protected]} INBOX

mailutil is from the UW-IMAP project. Every Linux distro should contain it.

When it failed (which it didn't) I would just go into the exchange account and delete "INBOX"

4
  • This is a nice try, but it won't work for a few reasons. The first is, as I mentioned, we really can't delete the box on '07. The second, as I also mentioned, is that we don't want end-users doing this by hand, it will take forever (400 messages is a 5-10 minute proposition right now). I haven't even mentioned how Exchange does (not) handle email with corrupted attachments...still, nice try. May 15, 2009 at 0:00
  • (wish the comments would be expanded) I will give mailutil a try tomorrow some time. May 15, 2009 at 0:03
  • Well I did not have to start over with any user but the first when I did it. mailutil does not create new headers. You can use multiple ssh-sessions (use screen!) to increase the performance maybe. Your folder names should not be an issue, mailutil will read the maildir (over imap...) and Exchange can surely allow any folder name and kind of hierachie (e.g. mails inside folder vs. folder with mail an folder with folders as with mbox). It worked for me for 30 users and went fast. We did it user by user, not over the weekend. mailutil transfer will not delete the old mailbox,so give it a try!
    – Christian
    May 15, 2009 at 0:20
  • I ended up with a different utility, although the results were acceptable. You received the answer vote because your description was closest to the approach I took. For the others, I do thank you for your input, and it was valuable; you have been upvoted accordingly. May 24, 2009 at 5:41
4

I've done many such migrations on much larger systems; 300+ mailboxes and I've migrated in many directions: Courier -> Cyrus, Exchange -> Cyrus, Courier -> Exchange, and Cyrus -> Exchange...

Our tool of choice is called imapsync; an open source Perl script. It uses the actual IMAP protocol to handle migrations thus alleviating the need to deal with underlying specifics to each implementation (aildir formats, mailbox annotations such as DONT (.) vs. slash (/) seperators.

IMAPSYNC is also re-entrant safe. You can run it multiple times over the same accounts and it will only copy what has not been copied the first time through or any new emails that may have arrived.

We typically dump all the user accounts to a BASH script that, in the end, is executed like so:

./imapsync --host1 mail.pozicom.net --user1 someone@domain --password1 mypassword \
           --host2 localhost --user2 someone@domain --password2 mypassword

Where host1 is where you're copying from host2 is where you're copying to.

You can get IMAPSYNC from here: http://freshmeat.net/projects/imapsync/

1

I've used the "imapsync" program before to mmigrate from an IMAP server to Scalix (an Exchange-like system). Think rsync for IMAP mailboxes.

Jim.

1

We have a huge large mail system, and routinely ingest users from other mailsystems (typically Notes or Groupwise). One common thread in all of these migrations is that the data migration piece is always a disaster. There are always intractable problems, VIP issues end up taking priority and the normal user population suffers. Migration tools have bugs that never get fixed because nobody does a Notes->Exchange migration twice!

What we're looking at for a future strategy is a mostly greenfield mailbox, with users identifying email that must be retained. For you its easy -- keep the dovecot servers running for 6 months, setup Outlook profiles pointing to the old imap system (I believe this is scriptable with Outlook 2007) and let users migrate in their own time.

You'll find that most folks really don't need what they say they need, and after 6 months, any legit need to move data to the new system will be done.

Another option is to use a mail archiving solution. We have to archive all email due to litigation, and oftentimes you can "ingest" messages from an old system into a hosted archiving solution. Check out Postini and Microsoft Hosted services.

0

Use the Transporter Suite. It was written specifically for this purpose.

Basically, you import a CSV file of the IMAP users and passwords, and it will import the data correctly. You can also import users one at a time if you are scared of doing a batch process. It's actually pretty hard to botch an import.

Do not use imapsync or any other IMAP to IMAP based sync engine. It sort of works, but much of the metadata will be screwed up -- specifically dates. Even with the --syncinternaldates option, Exchange will still overwrite the dates to the current date, and you will have a botched migration.

3
  • We ended up with an IMAP-based solution anyways. The Transporter Suite was not an option because of concerns on "reversability". And the dates on messages came out just fine. Jul 30, 2009 at 0:20
  • Glad they fixed the date bug then. What were the concerns on "reversability?" Aug 2, 2009 at 6:10
  • @Jesse Weigert, our Windows admin pointed out that once you had a box on the Exchange server, the current (2007) implementation wants to delete the user account as well as the user's box, which is a big problem. You can't just "delete the box" and start over when that user is actively using the account on the network. Sep 16, 2009 at 17:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .