httpd keeps crashing without any reference to why in the logs

Question

I have the logs set to debug in the hopes of tracking down what's causing the crash, but I can't find anything. Here is the error_log.

    [Thu Jan 06 10:27:35 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 19999 for (*)
[Thu Jan 06 14:47:04 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 06 14:47:04 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Thu Jan 06 14:47:04 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Thu Jan 06 14:47:04 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Thu Jan 06 14:47:04 2011] [info] Init: Initializing (virtual) servers for SSL
[Thu Jan 06 14:47:04 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Thu Jan 06 14:47:04 2011] [notice] Digest: generating secret for digest authentication ...
[Thu Jan 06 14:47:04 2011] [notice] Digest: done
[Thu Jan 06 14:47:04 2011] [debug] util_ldap.c(2021): LDAP merging Shared Cache conf: shm=0xb9dc2480 rmm=0xb9dc24b0 for VHOST: server.fredfinn.com
[Thu Jan 06 14:47:04 2011] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Thu Jan 06 14:47:04 2011] [info] LDAP: SSL support available
[Thu Jan 06 14:47:05 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Thu Jan 06 14:47:05 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Thu Jan 06 14:47:05 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(374): shmcb_init allocated 512000 bytes of shared memory
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(554): entered shmcb_init_memory()
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(576): for 512000 bytes, recommending 4266 indexes
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(619): shmcb_init_memory choices follow
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(621): division_mask = 0x1F
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(623): division_offset = 64
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(625): division_size = 15998
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(627): queue_size = 1604
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(629): index_num = 133
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(631): index_offset = 8
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(633): index_size = 12
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(635): cache_data_offset = 8
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(637): cache_data_size = 14386
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(650): leaving shmcb_init_memory()
[Thu Jan 06 14:47:05 2011] [info] Shared memory session cache initialised
[Thu Jan 06 14:47:05 2011] [info] Init: Initializing (virtual) servers for SSL
[Thu Jan 06 14:47:05 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Thu Jan 06 14:47:05 2011] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26527 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26527 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26528 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26528 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26529 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26529 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26530 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26530 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26532 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26532 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26533 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26533 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26534 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26534 for (*)
[Thu Jan 06 14:47:05 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Thu Jan 06 14:47:05 2011] [info] Server built: Aug 30 2010 12:32:08
[Thu Jan 06 14:47:05 2011] [debug] prefork.c(991): AcceptMutex: sysvsem (default: sysvsem)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26531 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26531 for (*)

The logs are setup as:

ErrorLog logs/error_log

LogLevel debug

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

CustomLog logs/access_log common

CustomLog logs/access_log combined

ServerSignature On

how many virtual hosts do you have (if any) and how is the logging setup ? — silviud, Jan 6, 2011 at 23:33
I had an issue that may be similar - question: are you running your httpd binary from the location it was compiled to be run from? — PP., Jan 7, 2011 at 8:17

JOTN · Accepted Answer · 2011-01-06 23:54:51Z

1

Are you getting a core dump file? You can run the debugger on it to see where it's crashing. That will let you ID if the crash is in the apache core code or in one of the modules. If it's a module then you can look for bugs and updates for that module.

See this for setting where it tries to dump core:

http://httpd.apache.org/docs/2.0/mod/mpm_common.html#coredumpdirectory

answered Jan 6, 2011 at 23:54

JOTN

1,7571 gold badge10 silver badges12 bronze badges

Ok cool! I set that in the httpd to the log dir. Hopefully that will give me something to work with... Thanks!
– Fred
Jan 7, 2011 at 0:19
"You can run the debugger on it to see where it's crashing" which debugger are you referring to?
– Derek Downey
Jan 7, 2011 at 1:20
1

I think he means gdb.
– Christian
Jan 7, 2011 at 7:36
Yes, gdb, dbx, or whatever is on you system: gdb /path/to/httpd /path/to/core
– JOTN
Jan 7, 2011 at 13:03

Add a comment |

Christian · Accepted Answer · 2011-01-07 07:43:56Z

0

Another method would be to run apache in foreground and additionally start it with strace.

First attempt:

httpd -X -f <your config file>

Second attempt:

strace httpd -X -f <your config file>

This will give you alot of output, from apache and the system calls, but perhaps you'll find something interesting. With strace -o <filename> you can write the strace output to a file which makes post processing much easier.

answered Jan 7, 2011 at 7:43

Christian

4,7232 gold badges26 silver badges27 bronze badges

Add a comment |

Fred · Accepted Answer · 2011-01-18 21:26:14Z

0

Turns out mysql was eating up TONS of ram and sending me over the limit for the VPS and consequentially killing the httpd. I was running wordpress. I've removed it this morning, and the ram is back to a reasonable level. We'll see if the crashing continues. Thanks for all your help, I learned a lot!

answered Jan 18, 2011 at 21:26

Fred

1431 silver badge8 bronze badges

Goodbye wordpress. That solved it.
– Fred
Jan 20, 2011 at 16:50

Add a comment |

Stack Exchange Network

httpd keeps crashing without any reference to why in the logs

3 Answers 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
apache-2.2
httpd
centos5
.

Linked

Hot Network Questions

httpd keeps crashing without any reference to why in the logs

3 Answers 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged apache-2.2httpdcentos5.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
apache-2.2
httpd
centos5
.