Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have the logs set to debug in the hopes of tracking down what's causing the crash, but I can't find anything. Here is the error_log.

    [Thu Jan 06 10:27:35 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 19999 for (*)
[Thu Jan 06 14:47:04 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 06 14:47:04 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Thu Jan 06 14:47:04 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Thu Jan 06 14:47:04 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Thu Jan 06 14:47:04 2011] [info] Init: Initializing (virtual) servers for SSL
[Thu Jan 06 14:47:04 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Thu Jan 06 14:47:04 2011] [notice] Digest: generating secret for digest authentication ...
[Thu Jan 06 14:47:04 2011] [notice] Digest: done
[Thu Jan 06 14:47:04 2011] [debug] util_ldap.c(2021): LDAP merging Shared Cache conf: shm=0xb9dc2480 rmm=0xb9dc24b0 for VHOST: server.fredfinn.com
[Thu Jan 06 14:47:04 2011] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Thu Jan 06 14:47:04 2011] [info] LDAP: SSL support available
[Thu Jan 06 14:47:05 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Thu Jan 06 14:47:05 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Thu Jan 06 14:47:05 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(374): shmcb_init allocated 512000 bytes of shared memory
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(554): entered shmcb_init_memory()
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(576): for 512000 bytes, recommending 4266 indexes
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(619): shmcb_init_memory choices follow
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(621): division_mask = 0x1F
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(623): division_offset = 64
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(625): division_size = 15998
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(627): queue_size = 1604
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(629): index_num = 133
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(631): index_offset = 8
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(633): index_size = 12
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(635): cache_data_offset = 8
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(637): cache_data_size = 14386
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(650): leaving shmcb_init_memory()
[Thu Jan 06 14:47:05 2011] [info] Shared memory session cache initialised
[Thu Jan 06 14:47:05 2011] [info] Init: Initializing (virtual) servers for SSL
[Thu Jan 06 14:47:05 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Thu Jan 06 14:47:05 2011] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26527 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26527 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26528 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26528 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26529 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26529 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26530 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26530 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26532 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26532 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26533 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26533 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26534 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26534 for (*)
[Thu Jan 06 14:47:05 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Thu Jan 06 14:47:05 2011] [info] Server built: Aug 30 2010 12:32:08
[Thu Jan 06 14:47:05 2011] [debug] prefork.c(991): AcceptMutex: sysvsem (default: sysvsem)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26531 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26531 for (*)

The logs are setup as:

ErrorLog logs/error_log

LogLevel debug

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

CustomLog logs/access_log common

CustomLog logs/access_log combined

ServerSignature On
share|improve this question
    
how many virtual hosts do you have (if any) and how is the logging setup ? –  silviud Jan 6 '11 at 23:33
    
Added log setup to bottom of post. –  Fred Jan 6 '11 at 23:40
    
I had an issue that may be similar - question: are you running your httpd binary from the location it was compiled to be run from? –  PP. Jan 7 '11 at 8:17
add comment

3 Answers

Are you getting a core dump file? You can run the debugger on it to see where it's crashing. That will let you ID if the crash is in the apache core code or in one of the modules. If it's a module then you can look for bugs and updates for that module.

See this for setting where it tries to dump core:

http://httpd.apache.org/docs/2.0/mod/mpm_common.html#coredumpdirectory

share|improve this answer
    
Ok cool! I set that in the httpd to the log dir. Hopefully that will give me something to work with... Thanks! –  Fred Jan 7 '11 at 0:19
    
"You can run the debugger on it to see where it's crashing" which debugger are you referring to? –  DTest Jan 7 '11 at 1:20
1  
I think he means gdb. –  Christian Jan 7 '11 at 7:36
    
Yes, gdb, dbx, or whatever is on you system: gdb /path/to/httpd /path/to/core –  JOTN Jan 7 '11 at 13:03
add comment

Another method would be to run apache in foreground and additionally start it with strace.

First attempt:

httpd -X -f <your config file>

Second attempt:

strace httpd -X -f <your config file>

This will give you alot of output, from apache and the system calls, but perhaps you'll find something interesting. With strace -o <filename> you can write the strace output to a file which makes post processing much easier.

share|improve this answer
add comment
up vote 0 down vote accepted

Turns out mysql was eating up TONS of ram and sending me over the limit for the VPS and consequentially killing the httpd. I was running wordpress. I've removed it this morning, and the ram is back to a reasonable level. We'll see if the crashing continues. Thanks for all your help, I learned a lot!

share|improve this answer
    
Goodbye wordpress. That solved it. –  Fred Jan 20 '11 at 16:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.