1

So this started happening for no apparent reason on a RHEL5 server running Plesk 10. I woke up one morning to find all the sites hosted on this box were offline. I SSH'd in and restarted httpd -

Stopping httpd:                                            [  OK  ]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

Alright, so I

ps ax | grep http
kill (the pid)
service httpd start

And everything boots up nicely.. For less than 24 hours, before it happens at the exact same time again. After I 'fixed' it that time, it went 13 days before crashing again, at the exact same time. So to reiterate - the httpd service (I assume) restarted and failed at the following times: 4/27/2012 04:13:52, 4/14/2012 04:14:18, 4/13/2012 04:12:48

I checked my cron logs and found the following entries that I don't understand and Google has failed me. They occur right around the time of the crash, and only occur on the days that the server has crashed... Too much to be a coincidence I think.

Suspicious/Confusing cron log entries

[me@www httpd]# cat ../cro* | grep RELOAD
Apr 13 09:40:01 www crond[5322]: (myusername2) RELOAD (cron/myusername2)
Apr 14 04:13:01 www crond[5322]: (myusername2) RELOAD (cron/myusername2)
Apr 14 14:27:01 www crond[5322]: (myusername2) RELOAD (cron/myusername2)
[me@www httpd]# cat ../cro* | grep LIST
Apr 27 04:13:14 www crontab[12973]: (root) LIST (myusername1)
Apr 13 04:12:09 www crontab[30867]: (root) LIST (myusername2)
Apr 13 09:39:57 www crontab[8274]: (root) LIST (myusername2)
Apr 14 04:13:01 www crontab[12193]: (root) LIST (myusername2)
Apr 14 14:26:09 www crontab[27898]: (root) LIST (myusername2)
[me@www httpd]# cat ../cro* | grep REPLACE
Apr 27 04:13:14 www crontab[12974]: (root) REPLACE (myusername1)
Apr 13 04:12:09 www crontab[30868]: (root) REPLACE (myusername2)
Apr 13 09:39:57 www crontab[8275]: (root) REPLACE (myusername2)
Apr 14 04:13:01 www crontab[12194]: (root) REPLACE (myusername2)
Apr 14 14:26:09 www crontab[27899]: (root) REPLACE (myusername2)

Cron logs from -5 mins of crash

4/27/2012 crash, 04:13:52

Apr 27 04:10:01 www crond[5189]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Apr 27 04:10:01 www crond[5192]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 27 04:10:01 www crond[5193]: (psaadm) CMD (/usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1)
Apr 27 04:10:01 www crond[5195]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:10:01 www crond[5196]: (root) CMD (php /path/to/pimcore/cli/maintenance.php)
Apr 27 04:10:01 www crond[5198]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:10:01 www crond[5200]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:11:01 www crond[5711]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:12:01 www crond[6152]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 27 04:12:01 www crond[6154]: (root) CMD (lynx -dump http://www.domain.com/script
Apr 27 04:13:14 www crontab[12973]: (root) LIST (myusername1)
Apr 27 04:13:14 www crontab[12974]: (root) REPLACE (myusername1)

4/14/2012 crash, 04:14:18

Apr 14 04:10:01 www crond[4712]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Apr 14 04:10:01 www crond[4716]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 14 04:10:01 www crond[4718]: (psaadm) CMD (/usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1)
Apr 14 04:10:01 www crond[4720]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:10:01 www crond[4721]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:10:01 www crond[4722]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:10:01 www crond[4724]: (root) CMD (php /path/to/pimcore)
Apr 14 04:11:01 www crond[5190]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:12:01 www crond[5543]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:12:01 www crond[5545]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:13:01 www crontab[12193]: (root) LIST (user)
Apr 14 04:13:01 www crontab[12194]: (root) REPLACE (myusername2)
Apr 14 04:13:01 www crond[5322]: (myusername2) RELOAD (cron/myusername2)
Apr 14 04:14:01 www crond[13896]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 14 04:14:01 www crond[13897]: (root) CMD (lynx -dump http://www.domain.com/script)

4/13/2012 crash, 04:12:48

Apr 13 04:10:01 www crond[23751]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Apr 13 04:10:01 www crond[23754]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:10:01 www crond[23755]: (psaadm) CMD (/usr/local/psa/admin/bin/php /opt/plesk-billing/admin/sbin/runevents.php > /dev/null 2>&1)
Apr 13 04:10:01 www crond[23756]: (root) CMD (/usr/lib/sa/sa1 1 1)
Apr 13 04:10:01 www crond[23758]: (root) CMD (php /path/to/pimcore)
Apr 13 04:10:01 www crond[23760]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:10:01 www crond[23761]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:11:01 www crond[24126]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:12:01 www crond[26995]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:12:01 www crond[26996]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:12:09 www crontab[30867]: (root) LIST (myusername2)
Apr 13 04:12:09 www crontab[30868]: (root) REPLACE (myusername2)
Apr 13 04:14:01 www crond[799]: (root) CMD (lynx -dump http://www.domain.com/script)
Apr 13 04:14:01 www crond[800]: (root) CMD (lynx -dump http://www.domain.com/script)

So as a lowtech temporary solution I've been setting my alarm for 4:15 AM every morning to check if my sites are down. Please help me out, I'm literally losing sleep over it. Thank you.

Edit 1:

/var/spool/cron/myusername1 AND /var/spool/cron/myusername2

Are both empty, if it matters. They are both also trusted (by my standards) users.

Edit 2:

Just noticed the following in /var/log/messages*

Apr 27 04:13:11 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 27 04:13:14 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 13 04:12:08 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 13 04:12:08 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 14 04:12:59 www named[3541]: max open files (1024) is smaller than max sockets (4096)
Apr 14 04:13:00 www named[3541]: max open files (1024) is smaller than max sockets (4096)

I wish I was a real admin, maybe I could understand this better. Not sure if it's relevant or not, but since it's the only other message in my logs I can find that is closely related to the server crash dates/times I'm including it.

5
  • 1
    This is awesome that you have an alarm dedicated to fixing your server. Perhaps run a cron entry to strace() your Apache process for 2 minutes at 4:12?
    – atx
    Apr 30, 2012 at 3:25
  • 1
    One man's awesome is another man's migraine Apr 30, 2012 at 3:27
  • What is the output of uname -a? Try updating bind for the last problem: cyberciti.biz/faq/max-open-files-1024-smaller-than-max-sockets
    – atx
    Apr 30, 2012 at 3:30
  • Linux (my revealing hostname) 2.6.18-274.el5PAE #1 SMP Fri Jul 8 17:59:09 EDT 2011 i686 i686 i386 GNU/Linux Apr 30, 2012 at 3:33
  • That is an incredibly old kernel. Do "yum update kernel" and reboot, and you should be fine Apr 30, 2012 at 5:00

0

You must log in to answer this question.

Browse other questions tagged .