So this started happening for no apparent reason on a RHEL5 server running Plesk 10. I woke up one morning to find all the sites hosted on this box were offline. I SSH'd in and restarted httpd -
Stopping httpd: [ OK ]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
Alright, so I
ps ax | grep http
kill (the pid)
service httpd start
And everything boots up nicely.. For less than 24 hours, before it happens at the exact same time again. After I 'fixed' it that time, it went 13 days before crashing again, at the exact same time. So to reiterate - the httpd service (I assume) restarted and failed at the following times: 4/27/2012 04:13:52, 4/14/2012 04:14:18, 4/13/2012 04:12:48
I checked my cron logs and found the following entries that I don't understand and Google has failed me. They occur right around the time of the crash, and only occur on the days that the server has crashed... Too much to be a coincidence I think.
Suspicious/Confusing cron log entries
[me@www httpd]# cat ../cro* | grep RELOAD
Apr 13 09:40:01 www crond[5322]: (myusername2) RELOAD (cron/myusername2)
Apr 14 04:13:01 www crond[5322]: (myusername2) RELOAD (cron/myusername2)
Apr 14 14:27:01 www crond[5322]: (myusername2) RELOAD (cron/myusername2)
[me@www httpd]# cat ../cro* | grep LIST
Apr 27 04:13:14 www crontab[12973]: (root) LIST (myusername1)
Apr 13 04:12:09 www crontab[30867]: (root) LIST (myusername2)
Apr 13 09:39:57 www crontab[8274]: (root) LIST (myusername2)
Apr 14 04:13:01 www crontab[12193]: (root) LIST (myusername2)
Apr 14 14:26:09 www crontab[27898]: (root) LIST (myusername2)
[me@www httpd]# cat ../cro* | grep REPLACE
Apr 27 04:13:14 www crontab[12974]: (root) REPLACE (myusername1)
Apr 13 04:12:09 www crontab[30868]: (root) REPLACE (myusername2)
Apr 13 09:39:57 www crontab[8275]: (root) REPLACE (myusername2)
Apr 14 04:13:01 www crontab[12194]: (root) REPLACE (myusername2)
Apr 14 14:26:09 www crontab[27899]: (root) REPLACE (myusername2)
Cron logs from -5 mins of crash
4/27/2012 crash, 04:13:52
Apr 27 04:10:01 www crond[5189]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Apr 27 04:10:01 www crond[5192]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 27 04:10:01 www crond[5193]: (psaadm) CMD (/usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1)
Apr 27 04:10:01 www crond[5195]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:10:01 www crond[5196]: (root) CMD (php /path/to/pimcore/cli/maintenance.php)
Apr 27 04:10:01 www crond[5198]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:10:01 www crond[5200]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:11:01 www crond[5711]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:12:01 www crond[6152]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:12:01 www crond[6154]: (root) CMD (lynx -dump http://www.domain.com/script
Apr 27 04:13:14 www crontab[12973]: (root) LIST (myusername1)
Apr 27 04:13:14 www crontab[12974]: (root) REPLACE (myusername1)
4/14/2012 crash, 04:14:18
Apr 14 04:10:01 www crond[4712]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Apr 14 04:10:01 www crond[4716]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 14 04:10:01 www crond[4718]: (psaadm) CMD (/usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1)
Apr 14 04:10:01 www crond[4720]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:10:01 www crond[4721]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:10:01 www crond[4722]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:10:01 www crond[4724]: (root) CMD (php /path/to/pimcore)
Apr 14 04:11:01 www crond[5190]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:12:01 www crond[5543]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:12:01 www crond[5545]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:13:01 www crontab[12193]: (root) LIST (user)
Apr 14 04:13:01 www crontab[12194]: (root) REPLACE (myusername2)
Apr 14 04:13:01 www crond[5322]: (myusername2) RELOAD (cron/myusername2)
Apr 14 04:14:01 www crond[13896]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:14:01 www crond[13897]: (root) CMD (lynx -dump http://www.domain.com/script)
4/13/2012 crash, 04:12:48
Apr 13 04:10:01 www crond[23751]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Apr 13 04:10:01 www crond[23754]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:10:01 www crond[23755]: (psaadm) CMD (/usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1)
Apr 13 04:10:01 www crond[23756]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 13 04:10:01 www crond[23758]: (root) CMD (php /path/to/pimcore)
Apr 13 04:10:01 www crond[23760]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:10:01 www crond[23761]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:11:01 www crond[24126]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:12:01 www crond[26995]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:12:01 www crond[26996]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:12:09 www crontab[30867]: (root) LIST (myusername2)
Apr 13 04:12:09 www crontab[30868]: (root) REPLACE (myusername2)
Apr 13 04:14:01 www crond[799]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:14:01 www crond[800]: (root) CMD (lynx -dump http://www.domain.com/script)
So as a lowtech temporary solution I've been setting my alarm for 4:15 AM every morning to check if my sites are down. Please help me out, I'm literally losing sleep over it. Thank you.
Edit 1:
/var/spool/cron/myusername1 AND /var/spool/cron/myusername2
Are both empty, if it matters. They are both also trusted (by my standards) users.
Edit 2:
Just noticed the following in /var/log/messages*
Apr 27 04:13:11 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 27 04:13:14 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 13 04:12:08 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 13 04:12:08 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 14 04:12:59 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 14 04:13:00 www named[3541]: max open files (1024) is smaller than max sockets (4096)
I wish I was a real admin, maybe I could understand this better. Not sure if it's relevant or not, but since it's the only other message in my logs I can find that is closely related to the server crash dates/times I'm including it.