I'm looking for an imap server that is fast with larger folders. Say 20'000-100'000 emails per folder.

Currently I'm using dovecot, and opening a folder can take 10 seconds, and the HD light on the imap server is brinking like crazy.

I'm using alpine as a client, and it only lists the newest mails by default, so it's not that my client is trying to transfer everything when opening the mailbox. This can be seen in that when I scroll it has to load the subject lines for the next page (the first time I scroll there).

I'm using maildir on XFS.

Edit: I ask since it's not much data, in the grand scheme of things. If this was in a SQL database then getting the subject lines of the newest 40 messages would not take 10 seconds for a folder of 40'000 emails. The only data needed is:

SELECT date, from, subject FROM emails ORDER BY date DESC LIMIT 40;

Any ideas?

  • I would suggest splitting these folders up. It will be nicer for the user and for the server. – alex Nov 26 '09 at 22:53
  • I'm the user, and no it wouldn't be. It's the ugly fix I'm going with though. – Thomas Nov 27 '09 at 17:35

Dovecot is actually pretty good, performance-wise. Dovecot's Performance Tuning wiki page has a few tips and tricks to further improve performance. Keeping indices and maildirs on separate disks is a good thing to start with, if at all possible for you. You could also evaluate switching to Dovecot's dbox storage format.

  • I've tried to optimize using the tuning page. This is as good as I got it. I'll try dbox. – Thomas Aug 24 '09 at 12:31
  • dovecot is bonkers fast – hendry Feb 2 '12 at 16:18

Maybe you could try using a database engine for message storage, instead of using Maildir/Maildir++ mailboxes. This can be done with dbmail.

I don't know how reliable dbmail is for a production environment, but since you already have virtualization working, you could set it up on another VM for testing purposes and see how it performs on your environment.

Here's an overview of dbmail's architecture:

alt text
(source: dbmail.org)

  • I don't think that the graphic adds any value to the otherwise nice answer. There's also an interesting thread on the dovecot mailing list, benchmarking some aspects of dovecot vs dbmail (dovecot.org/list/dovecot/2007-May/022599.html). – earl Aug 24 '09 at 17:42
  • Earl, thanks for the feedback. I think that the OP mileage might vary, and that's why it would be useful for him to perform some tests on his own environment. It's not my intention to question the value of the benchmarks that you linked, but since those tests were run 2.5 years (and 6 releases) ago, it wouldn't be a bad idea to check again. – mfriedman Aug 24 '09 at 18:29
  • Oh, and regarding the diagram that I've added to my answer, I respectfully disagree. It shows how the pieces fit together, what are the DBMail and the Non-DBMail components, and how they interact with each other. Even when I agree that the picture may be beyond what was asked, I think that it helps to illustrate how this particular solution works. And, as the saying goes, "a picture is worth a thousand words". Best regards, Marcus. – mfriedman Aug 24 '09 at 18:35
  • Thank you for the link and diagram. I have done some subjective "feel" benchmarking and it seems that dovecot is slightly faster once it has cached the index, but since emails keep trickling in the cache is soon invalidated so in practice I get the slow hit every time I open the mailbox. dbmail seems to take the hit incrementally on email insertion (makes sense), so it's while it may not be quite as fast, I never get the 10s wait to open a folder. Tested with 44'000 emails in a box. – Thomas Aug 24 '09 at 21:25

You don't mention the server specs...how much memory are you using, processor, network card/switches are gigabit? And if you look at the server, can you tell what's being maxed out? If it's drive throughput you aren't going to get very far changing the server software

I've been cloning systems over a network and was puzzled to have two systems on a gigabit switch pulling only about 15 MB/sec when I knew that my system was capable of bursts in the 50 MB/s range. Turned out it was the disks bottlenecking on the end-systems (I put a drive into a second IDE channel and did a direct DD, got the same transfer speeds).

You might want to check out the processor/disk/network usage as well as the switch and see if any of those are causing issues. If not those, you could look for ways to boost throughput using separate disks, separating mailboxes to different spindles, check and see if you can get any better throughput using hardware RAID mirroring (I'm not sure how much of a boost in read times off the disks you can get), or possibly moving to higher-performing hard disks with lower latency and bigger caches.

  • My point is that it's not much data. 40'000 entries is nothing to a database. It shouldn't be much for an imap server. It's a virtual machine (KVM on Linux) on a quad-core 2.4GHz with raid5 and 8GB RAM, but I've seen this on a physical machine as well. I've benchmarked the disks and network and I get about than 100MByte/sec. But remember that all this is just reading select mail headers (from, flags, subject) of about 40 (the latest) of 40'000 emails. Not even the 40 mail bodies. – Thomas Aug 24 '09 at 12:22
  • And not 40'000 mail headers. They are loaded on demand if I scroll. – Thomas Aug 24 '09 at 12:23
  • You probably already checked this (nice specs on the server...shouldn't be that is the issue, I'd think) but is it possibly the client system reading the locally cached information and going back and forth that is causing some slowdown? – Bart Silverstrim Aug 24 '09 at 12:35

Since you're using dovecot I presume you're already using it's indexing features? I'm not aware of anything (at least, not anything free), that's faster than dovecot.

  • Yes, I'm using the indexing. The thing that confuses me is that an SQL database would not take 10 seconds to list the newest 40 rows (when properly indexed). I've updated the question with this. – Thomas Aug 24 '09 at 12:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.