I had one of our Apache httpd servers "die" in a weird way overnight. I have looked at all relevant logs and cannot find anything out of the ordinary.
This happened in our development environment thankfully, but I am rather concerned because our dev environment nearly identically mirrors our production environment.
Can anyone tell me what could have happened here?
Symptoms & Troubleshooting Steps
- Apache httpd was completely unresponsive
- Load average of 0.0 across the board
- No log anomalies
ps aux | grep httpd
appeared normaliptables -L
showed port 80 open- SELinux is enabled and properly configured: no AVCs logged
- No configuration changes were made to any system components, no matter how insignificant
- Apache httpd was reloaded at
17:11:25
server time to clear APC (php) cache- Successful requests were made after this reload through
17:20
after which I left for the day - Nothing remarkable happened after reload
- Successful requests were made after this reload through
- The following morning the httpd processes were unresponsive
- Telnet to port 80 connected successfully
- Manual GET, POST, OPTIONS, and HEAD requests received no response whatsoever; nothing logged
Logs
access_log
172.22.44.41 - - [03/Mar/2011:17:20:18 -0600] "GET /admin.php?type=notifications HTTP/1.0" 200 44326 "http://umc.dev.example.com/admin.php" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.107 Safari/534.13"
172.22.44.41 - - [04/Mar/2011:14:45:16 -0600] "GET / HTTP/1.0" 200 62921 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.107 Safari/534.13"
error_log
[Thu Mar 03 17:11:25 2011] [notice] SIGHUP received. Attempting to restart
[Thu Mar 03 17:11:25 2011] [notice] Digest: generating secret for digest authentication ...
[Thu Mar 03 17:11:25 2011] [notice] Digest: done
[Thu Mar 03 17:11:25 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Fri Mar 04 14:45:05 2011] [notice] caught SIGTERM, shutting down
[Fri Mar 04 14:45:05 2011] [notice] SELinux policy enabled; httpd running as context user_u:system_r:httpd_t:s0
[Fri Mar 04 14:45:05 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 04 14:45:05 2011] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 04 14:45:05 2011] [notice] Digest: done
[Fri Mar 04 14:45:05 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
ps aux | grep httpd
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 8477 0.0 1.9 408748 9876 ? Ss Feb24 0:03 /usr/sbin/httpd
apache 19933 0.4 14.6 436436 74692 ? S Mar03 0:01 /usr/sbin/httpd
apache 19936 0.4 8.4 419708 43148 ? S Mar03 0:01 /usr/sbin/httpd
/var/log/messages
Feb 27 04:02:01 webdev1 syslogd 1.4.1: restart.
Mar 4 14:43:35 webdev1 yum: Installed: strace-4.5.18-5.el5_5.5.x86_64
Versions
OS & Kernel
- CentOS release 5.5 (Final)
- Linux webdev1.int.example.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
Apache httpd
Server version: Apache/2.2.3
Server built: Aug 30 2010 12:28:40
Server's Module Magic Number: 20051115:3
Server loaded: APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
PHP
- php53u-5.3.5-3.ius.el5
Repositories used
CentOS Base & Updates for all packages with the following exception: php53u from IUS Community repository.