I have encountered this twice, the first time I thought it was just random stuff, so I put the investigation on hold until I encountered this again today.
Summary:
- Centos 5.7, sendmail 8.13.8
- High CPU usage by crond and sendmail (between 25% to 100% each)
- super slow disk IO (ssh runs fine until there is disk IO ops, like a simple ls -la)
- 'df -h' gave me a disk usage of 900G on my main mount (about 50% of my total space), I did not manage to track down which folder was using all the space, as the disk was running really slowly (it took 30s to return a 17M usage for one folder)
What I tried to do:
- Kill sendmail, crond, and any rogue PHP processes that was consuming more than the normal amount of CPU
- shutdown httpd, sendmail, mysqld (all of those took 1-5min to shutdown)
Results:
- Cannot kill sendmail process, it keeps appearing at the top with a TIME of 600Minutes+ (I only discovered that there was a problem when I woke up)
None of those worked, then suddenly out of the blue while I was just scanning through logs (very slowly...) it went back to the normal speed again.
a du -h revealed that the 900G of usage is gone and back to 19G (which is what it is supposed to be at), and the sendmail process is killable now...
I am not sure where to start with to investigate this wierd problem, so if anyone can share some tips or shed some light on possible causes thast would be fab!
EDIT (2011-11-01):
The only record I found of the process I killed was this one line
Oct 31 11:43:08 localhost sendmail[23302]: p9UIT2SB023302: from=[removed, it was not an email address though], size=1152544132509, class=0, nrcpts=1, relay=[removed]@localhost
As for cronjobs, I have 2 currently running, both are php scripts which I run once a minute and once an hour respectively.
Edit 2 (2011-11-01):
Wait a sec... size is 1152544132509 (about 1TB)... I guess that's what took the space... Now I gotta find what sent that mail. Any suggestions to how I can find the culprit?
Edit 3 (2011-11-03):
I found that I had over 10k unauthorized SSH login attempts... maybe the server was hijacked for a hacking attempt elsewhere... I will be switching off SSH login with password and making sure that the authorized keys are really the allowed ones...
I also installed denyhosts to auto block any further bruteforce attacks.
