Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got a bunch of duplicate messages in my IMAP server's Maildir. What's the best way to remove them?

Some relevant points:

  • Shared Message-ID is usually a good enough definition of duplicate. A tiny script that removes all but one of the duplicate messages would work.
  • Sometimes it's necessary to find duplicates based on shared message bodies. What's a reasonable definition of shared here? Bitwise equivalent? What about weird differences in line wrapping, escaping, character encoding?
  • Sometimes there's some meaningful difference between 'duplicate' messages. What's the best way to review the differences in sets of 'duplicate' messages? Diffs?
share|improve this question
add comment

3 Answers

for generic files in linux, I use fdupes utils to remove duplicate files. I found it also works for Maildir messages.

share|improve this answer
add comment

I've made some significant improvements to Kevin's script mentioned above, and he was kind enough to accept my pull requests. Eventually we split this off into a dedicated project which you can find here:

https://github.com/kdeldycke/maildir-deduplicate

share|improve this answer
    
This script moved to a new location so I updated the link. –  Adam Spiers Feb 3 at 12:32
add comment

Best I've found today is Kevin Deldycke's maildir-deduplicate.

  • It ignores the X-MIMETrack header by default and compares headers using the SHA224 digest.
  • It does not consider the message body at all.
  • It automatically deletes duplicates without asking for confirmation.
  • It assumes your mailbox is in /Users/kevin/Maildir. :)

I bet someone could make something fancy from Rick Sanders' delIMAPdups.pl, part of his IMAP Tools.

share|improve this answer
    
maildir-deduplicate moved to a new location so I updated the link. However your information is now out of date. –  Adam Spiers Feb 3 at 12:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.