53

Although I understand the basics of the two storage formats (1 file per email under Maildir vs. 1 single file per mailbox under mbox), I am wondering what the practical implications are here -

  • Is one storage format more scalable than the other?
  • Are there data integrity concerns / differences?
  • Are there clearly defined situations where you should use one format over the other?
0

7 Answers 7

22

Don't manage mailboxes from postfix. Never. Redirect messages for delivery via POP/IMAP server that has appropriate functionality. In case of dovecot there is dovecot-lda aka deliver that do everything and much more, like user-controlled message filtering, quota management, autoreplying and so on.

Anyway maildir is newer and preferrable format due to the lot of improvements comparatively to maibox. Maildir has an index for each folder that allow to control duplicates, expiration times and even full-text search. Also, maildir is significantly faster on a huge pile of messages. Dovecot can easily operate maildir with 300k messages in it without any visible slowdown. Mailbox such big is a problem itself. Also, most modern POP/IMAP servers has a lot of utilities for common tasks in large infrastructure.

20

If you're using NFS for mail storage, do not use mbox under any circumstances whatsoever. And if you want a scalable solution, Maildir is the way to go.

The main problem with the mbox format is that of file locking - if you have more than one mail server, or more than one process trying to access the mailbox at the same time, you run a large risk of getting a corrupted mailbox. It's also difficult to go through a mailbox and delete a large number of messages, for instance when you've suffered a bounce storm.

Maildir is designed to work without failure when you have several mail servers, or several processes on one server, delivering email to the same account at the same time as the IMAP or POP server is accessing the account.

The main problem with Maildir is if you are using a file system that slows down when handling too many inodes, and if your backup system is bad at handling multiple files. As for file systems, back when I did email sysadmining at an ISP, VXFS was the best for this. For backups, I don't have any recommendation; unfortunately most of them seem to be designed to handle databas servers rather than a gazillion small files.

9

To address the specific questions:

Is one storage format more scalable than the other?

mbox tends to become difficult to manage the bigger it gets. Since it stores everything in one big blob, it can be hard to backup incrementally, and such access will lock the mbox against writes while it happens. If there is some corruption, it's a lot harder to try to repair or recover - which you could say is a risk that increases as it stores more mail.

Maildir relies on the underlying filesystem for its scalability, as it will characteristically create many small files, one per message. If you have a lot of emails, Maildir is usually easier and faster to deal with.

Are there data integrity concerns / differences?

Yes, the main one is that mbox requires read/write file locking on the whole archive, meaning reads must wait for writes, and each individual write must wait for all other accesses. If it were used in such a way that locking isn't possible, such as over an NFS share, the archive is subject to corruption. Any random corruption can affect the whole archive rather than a single message.

With Maildir, locking can be at the level of the individual message, for most operations, and corruption in any file will only affect that message. Any indexes into the messages can be rebuilt if needed from the messages themselves.

Are there clearly defined situations where you should use one format over the other?

It's more the case that Maildir is starting to replace mbox as the universally preferred format, with no significant drawbacks.

Its main limitation is its creation of many, many small files, which had the ability to put a strain on some filesystems and/or be inefficient depending on filesystem cluster size. This is less of an issue with modern filesystems. It can also inform your backup strategy, but on the whole it does give more flexibility to your choice of backup strategy.

4

Postfix does not care, mostly. It just stuffs the mail into some file. The intersting part comes when you want to fetch the mail from the machine via IMAP (or POP3). Most IMAP-servers (I tried) prefer Maildirs, as these are easier extendable with various meta-data the IMAP-server could want, while still maintaining a basic "this is a Maildir" structure, so you can replace the IMAP server without breaking too much (you will obviously lose the meta-data, which also includes mail flags, so you should read migration docs of the IMAP server if you want to really switch).

3

Mbox is fast when searching a text inside a mailbox and appending a new mail into the mail file, but it's prone to mail corruption and may have problems with network file systems. Also, File locking may be an issue.

With Maildir, retrieving specific mails is faster, and has no issues due to file locking or network file systems (Example: With Maildir, you can store different mail folders in different NFS shares, this cannot be achieved with Mbox). You can have issues with Maildir when a filesystem is not efficient handling a large number of files.

I understand that you need an scalable solution, before you choose one or another, be sure to give a read to this document: (Even if you are not using Courier)

Benchmarking mbox versus maildir

1
  • 1
    To put things in perspective, that benchmark is primarily a benchmark between two different IMAP servers, and it's done by the author of the Courier IMAP server (which uses Maildir), and it was done in 2001 through 2003. The "high end computer" it was tested on was a Pentium-II era Celeron with 256MB of PC-100 SDRAM, with the ext2 filesystem. The assertions it makes about Maildir scalability, however, probably are even more true today than they were then. Oct 9, 2017 at 22:58
1

The main disadvantage I have run into with the maildir format is file store space used. Depending on the filesystem, each file is allocated space by blocks with a fixed number bytes per block. However, most email messages do not take up nearly the size of a block, and thus a ton of wasted space ensues. This holds true for directories in a lot of filesystem's directory structure -- each directory is allocated a whole block to hold the directory information. I find that using mailbox format saves me about half of the filesystem space.

That said, there is a case for end users to use a maildir format, but when archiving email on a server, use a mailbox format. And if possible, depending on the server settings, you can use compression on the mailbox files. This will make searches and retrievals a bit slow, but archived email is not for everyday access anyway.

2
  • 1
    Choosing appropriate block size and applying deduplication & compression on a mail storage, filesystem or block device level is available (and more easily composable) for maildir as well.
    – anx
    Jul 13, 2022 at 15:58
  • 1
    I would argue for archival purposes, speed&complexity of verifying/updating is even more important. An rsync backup is a backup where common practices can be applied, which most of the time will beat O(n) storage overhead.
    – anx
    Jul 13, 2022 at 16:02
-2

THE REAL ANSWER:

If you just check the logs...

cat /var/log/mail.warn

You will see that even the program will warn you:

Dec  1 18:47:25 example-hostname postfix/trivial-rewrite[2192]: warning: do not list domain exampledomain.eu in BOTH mydestination and virtual_mailbox_domains

This is also well documented on the official documentation page here (search for NEVER and you will imediately find the problem). So you should not do it if the programmers who created the program say not to. Period.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .