38

Although I understand the basics of the two storage formats (1 file per email under Maildir vs. 1 single file per mailbox under mbox), I am wondering what the practical implications are here -

  • Is one storage format more scalable than the other?
  • Are there data integrity concerns / differences?
  • Are there clearly defined situations where you should use one format over the other?

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

21

Don't manage mailboxes from postfix. Never. Redirect messages for delivery via POP/IMAP server that has appropriate functionality. In case of dovecot there is dovecot-lda aka deliver that do everything and much more, like user-controlled message filtering, quota management, autoreplying and so on.

Anyway maildir is newer and preferrable format due to the lot of improvements comparatively to maibox. Maildir has an index for each folder that allow to control duplicates, expiration times and even full-text search. Also, maildir is significantly faster on a huge pile of messages. Dovecot can easily operate maildir with 300k messages in it without any visible slowdown. Mailbox such big is a problem itself. Also, most modern POP/IMAP servers has a lot of utilities for common tasks in large infrastructure.

14

If you're using NFS for mail storage, do not use mbox under any circumstances whatsoever. And if you want a scalable solution, Maildir is the way to go.

The main problem with the mbox format is that of file locking - if you have more than one mail server, or more than one process trying to access the mailbox at the same time, you run a large risk of getting a corrupted mailbox. It's also difficult to go through a mailbox and delete a large number of messages, for instance when you've suffered a bounce storm.

Maildir is designed to work without failure when you have several mail servers, or several processes on one server, delivering email to the same account at the same time as the IMAP or POP server is accessing the account.

The main problem with Maildir is if you are using a file system that slows down when handling too many inodes, and if your backup system is bad at handling multiple files. As for file systems, back when I did email sysadmining at an ISP, VXFS was the best for this. For backups, I don't have any recommendation; unfortunately most of them seem to be designed to handle databas servers rather than a gazillion small files.

4

Postfix does not care, mostly. It just stuffs the mail into some file. The intersting part comes when you want to fetch the mail from the machine via IMAP (or POP3). Most IMAP-servers (I tried) prefer Maildirs, as these are easier extendable with various meta-data the IMAP-server could want, while still maintaining a basic "this is a Maildir" structure, so you can replace the IMAP server without breaking too much (you will obviously lose the meta-data, which also includes mail flags, so you should read migration docs of the IMAP server if you want to really switch).

4

To address the specific questions:

Is one storage format more scalable than the other?

mbox tends to become difficult to manage the bigger it gets. Since it stores everything in one big blob, it can be hard to backup incrementally, and such access will lock the mbox against writes while it happens. If there is some corruption, it's a lot harder to try to repair or recover - which you could say is a risk that increases as it stores more mail.

Maildir relies on the underlying filesystem for its scalability, as it will characteristically create many small files, one per message. If you have a lot of emails, Maildir is usually easier and faster to deal with.

Are there data integrity concerns / differences?

Yes, the main one is that mbox requires read/write file locking on the whole archive, meaning reads must wait for writes, and each individual write must wait for all other accesses. If it were used in such a way that locking isn't possible, such as over an NFS share, the archive is subject to corruption. Any random corruption can affect the whole archive rather than a single message.

With Maildir, locking can be at the level of the individual message, for most operations, and corruption in any file will only affect that message. Any indexes into the messages can be rebuilt if needed from the messages themselves.

Are there clearly defined situations where you should use one format over the other?

It's more the case that Maildir is starting to replace mbox as the universally preferred format, with no significant drawbacks.

Its main limitation is its creation of many, many small files, which had the ability to put a strain on some filesystems and/or be inefficient depending on filesystem cluster size. This is less of an issue with modern filesystems. It can also inform your backup strategy, but on the whole it does give more flexibility to your choice of backup strategy.

3

Mbox is fast when searching a text inside a mailbox and appending a new mail into the mail file, but it's prone to mail corruption and may have problems with network file systems. Also, File locking may be an issue.

With Maildir, retrieving specific mails is faster, and has no issues due to file locking or network file systems (Example: With Maildir, you can store different mail folders in different NFS shares, this cannot be achieved with Mbox). You can have issues with Maildir when a filesystem is not efficient handling a large number of files.

I understand that you need an scalable solution, before you choose one or another, be sure to give a read to this document: (Even if you are not using Courier)

Benchmarking mbox versus maildir

  • To put things in perspective, that benchmark is primarily a benchmark between two different IMAP servers, and it's done by the author of the Courier IMAP server (which uses Maildir), and it was done in 2001 through 2003. The "high end computer" it was tested on was a Pentium-II era Celeron with 256MB of PC-100 SDRAM, with the ext2 filesystem. The assertions it makes about Maildir scalability, however, probably are even more true today than they were then. – thomasrutter Oct 9 '17 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.