So... I see some very weird load issues on our e-mail server. It starts spiking around 8-9am (coincidentally that's when people start working), but it goes down around 11am or so. CPU usage remains normal, I have plenty of free memory, no swapping. Yesterday we had a really high iowait% (49.75) with a really high load (40), today we 'only' had a load of 11-12 with iowait% being between 3-4 tops.
All signs point to imapd as the culprit (courier-imap), because when I stopped it, load suddenly started to lower, and within 2-3 minutes, it was back to normal. I did have about 40-60 of them running. We use thunderbird, which opens 5 connections each, I lowered it to 1 on most workstations, it helped a bit (load went down to 5-7), then... the whole server went back to normal around 11am.
I still have ~30 imapds running, but with perfectly normal load (between 0.2 and 0.4). So... I don't really understand why is this happening, because, by logic, it should be much higher, if that would be the cause of the issue.
It's a Linode 1080 VPS with 1gig ram.
(chkrootkit / rkhunter showed nothing unusual.)