0

We are using apache http server 2.2.29 on Linux. Below is the information related to Linux version (as found in /proc/version file) Linux version 2.6.32-573.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015

We have observed that apache is generating many core dumps and getting crashed regularly. When it generates the core dump, below is being printed in the http’s error log:

[Mon Jun 20 01:28:09 2016] [notice] child pid 6520 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name [Mon Jun 20 03:27:54 2016] [notice] child pid 6462 exit signal Segmentation fault (11), possible coredump in /app/http_instance_name

Below core files had generated at these segmentation faults.

-rw------- 1 http http 65130496 Jun 20 01:28 core.6520 -rw------- 1 http http 66134016 Jun 20 03:27 core.6462

We are running on below Apache http version:

Server version: Apache/2.2.29 (Unix) Server built: Mar 4 2016 15:21:54

I have tried to analyse the core files using gdb tool , however, i was not able to understand much from it?

Please let me know if further information is required help get more understanding pertinent to this issue.

can this be due to some modules used in httpd.conf file? We are observing this issue across all our environments.

UPDATE: Common Pattern

The most common pattern I am observing when the segmentation fault error being is thrown is , the below error comes for most of the cases just before the segmentation fault appears

[Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rxxxx.xxxx.net:8543, referer: https://sit1-xxxxxx.co.uk:8443/CommonWeb/pages/common/jsp/login.jsp [Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/dojo.js, referer: https://sit1.xxxxxx:8443/CommonWeb/pages/common/jsp/login.jsp [Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] (-2)Unknown error 18446744073709551614: proxy: error reading status line from remote server rpovassxxxx.srxxx.net:8543, referer: https://sit1-****:8443/CommonWeb/pages/common/jsp/login.jsp [Mon May 16 16:08:02 2016] [error] [client 10.10.12.113] proxy: Error reading from remote server returned by /CommonWeb/pages/common/includes/js/resize.js, referer: https://xxxx-xxxx:8443/CommonWeb/pages/common/jsp/login.jsp [Mon May 16 16:08:03 2016] [notice] child pid 28308 exit signal Segmentation fault (11), possible coredump in /app/

Below is the output of 'gdb full' during debugging:

     #0  0x0000003704a0e7dd in read () from /lib64/libpthread.so.0
     No symbol table info available.
    #1  0x000000000044f8b7 in ap_mpm_pod_check (pod=<value optimized out>)     at pod.c:54
    c = <value optimized out>
    fd = 6
    rc = <value optimized out>
   #2  0x000000000044de74 in child_main (child_num_arg=1) at worker.c:1259
    threads = 0xc2bab0
    rv = <value optimized out>
    ts = 0xc079f8
    thread_attr = 0xc07a18
    start_thread_id = <value optimized out>
  #3  0x000000000044e0dd in make_child (s=0xb44870, slot=1) at worker.c:1342
    pid = 0
   #4  0x000000000044efc0 in perform_idle_server_maintenance     (_pconf=<value             optimized out>, plog=<value optimized out>, s=<value optimized out>)
at worker.c:1568
    j = <value optimized out>
    free_slots = {1, 5, 6, 7, 11817336, 0, 11784504, 0, -1236161560, 11137,   11817336, 0, 11800920, 0, 11815024, 0, 11784504, 0, -1361903789, 11137,
      4337480, 0, -1238262800, 11137, 29, 0, 12768544, 0, 11815024, 0, 12489224, 0}
    total_non_dead = <value optimized out>
    active_thread_count = <value optimized out>
    idle_thread_count = <value optimized out>
    free_length = 1
    i = <value optimized out>
    ps = <value optimized out>
    totally_free_length = <value optimized out>
    last_non_dead = <value optimized out>
      #5  server_main_loop (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at worker.c:1680
    status = 11
    pid = {pid = -1, in = 0xb45178, out = 0xb41158, err = 0xb3d138}
    i = <value optimized out>
    child_slot = <value optimized out>
    exitwhy = 6
    processed_status = <value optimized out>
    #6  ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>,     s=<value optimized out>) at worker.c:1782
    remaining_children_to_start = <value optimized out>
    rv = <value optimized out>
    #7  0x0000000000425e34 in main (argc=3, argv=0x7ffcafb44eb8) at main.c:753
    c = 0 '\000'
    configtestonly = <value optimized out>
    confname = 0x458a64 "conf/httpd.conf"
    def_server_root = 0x458a4f "/app/***"
    temp_error_log = 0x0
    error = <value optimized out>
    process = 0xb44870
    server_conf = 0xb44870
    pglobal = 0xb3b128
    pconf = 0xb3d138
    plog = 0xb45178
    ptemp = 0xb41158
    pcommands = 0xb3f148
    opt = 0xb3f238
    rv = <value optimized out>
    mod = <value optimized out>
    optarg = 0x0
    signal_server = <value optimized out>'

Below modules are being used in httpd.conf file.

   `cat httpd.conf | grep LoadMo

# have to place corresponding LoadModule' lines at this location so the # LoadModule foo_module modules/mod_foo.so LoadModule authn_file_module modules/mod_authn_file.so LoadModule authn_dbm_module modules/mod_authn_dbm.so LoadModule authn_anon_module modules/mod_authn_anon.so LoadModule authn_dbd_module modules/mod_authn_dbd.so LoadModule authn_default_module modules/mod_authn_default.so LoadModule authz_host_module modules/mod_authz_host.so LoadModule authz_groupfile_module modules/mod_authz_groupfile.so LoadModule authz_user_module modules/mod_authz_user.so LoadModule authz_dbm_module modules/mod_authz_dbm.so LoadModule authz_owner_module modules/mod_authz_owner.so LoadModule authz_default_module modules/mod_authz_default.so LoadModule auth_basic_module modules/mod_auth_basic.so LoadModule auth_digest_module modules/mod_auth_digest.so LoadModule dbd_module modules/mod_dbd.so LoadModule dumpio_module modules/mod_dumpio.so LoadModule reqtimeout_module modules/mod_reqtimeout.so LoadModule ext_filter_module modules/mod_ext_filter.so LoadModule include_module modules/mod_include.so LoadModule filter_module modules/mod_filter.so LoadModule substitute_module modules/mod_substitute.so LoadModule deflate_module modules/mod_deflate.so LoadModule log_config_module modules/mod_log_config.so LoadModule log_forensic_module modules/mod_log_forensic.so LoadModule logio_module modules/mod_logio.so LoadModule mime_magic_module modules/mod_mime_magic.so LoadModule cern_meta_module modules/mod_cern_meta.so LoadModule expires_module modules/mod_expires.so LoadModule headers_module modules/mod_headers.so LoadModule ident_module modules/mod_ident.so LoadModule usertrack_module modules/mod_usertrack.so LoadModule unique_id_module modules/mod_unique_id.so LoadModule setenvif_module modules/mod_setenvif.so LoadModule version_module modules/mod_version.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule proxy_ftp_module modules/mod_proxy_ftp.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_scgi_module modules/mod_proxy_scgi.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so #LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule ssl_module modules/mod_ssl.so LoadModule mime_module modules/mod_mime.so LoadModule dav_module modules/mod_dav.so LoadModule status_module modules/mod_status.so LoadModule autoindex_module modules/mod_autoindex.so LoadModule info_module modules/mod_info.so LoadModule cgid_module modules/mod_cgid.so LoadModule cgi_module modules/mod_cgi.so LoadModule dav_fs_module modules/mod_dav_fs.so LoadModule vhost_alias_module modules/mod_vhost_alias.so LoadModule negotiation_module modules/mod_negotiation.so LoadModule dir_module modules/mod_dir.so LoadModule imagemap_module modules/mod_imagemap.so LoadModule actions_module modules/mod_actions.so LoadModule speling_module modules/mod_speling.so LoadModule userdir_module modules/mod_userdir.so LoadModule alias_module modules/mod_alias.so LoadModule rewrite_module modules/mod_rewrite.so LoadModule auth_kerb_module modules/mod_auth_kerb.so LoadModule manager_module modules/mod_manager.so LoadModule slotmem_module modules/mod_slotmem.so LoadModule proxy_cluster_module modules/mod_proxy_cluster.so LoadModule advertise_module modules/mod_advertise.so

Below is the output of httpd -V (capital V)

 ./httpd -V
 Server version: Apache/2.2.29 (Unix)
 Server built:   Jan 13 2016 15:10:57
 Server's Module Magic Number: 20051115:36
 Server loaded:  APR 1.5.1, APR-Util 1.5.3
 Compiled using: APR 1.5.1, APR-Util 1.5.3
 Architecture:   64-bit
 Server MPM:     Worker
 threaded:     yes (fixed thread count)
 forked:     yes (variable process count)
 Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/app/****"
 -D SUEXEC_BIN="/app/****/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

**OUTPUT of file * **

     httpd.exp:              script text executable for .
     mod_actions.so:         ELF 64-bit LSB shared object, x86-64, version  1 (SYSV), dynamically linked, not stripped
     mod_advertise.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_alias.so:           ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_auth_basic.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_auth_digest.so:     ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_auth_kerb.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_authn_anon.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_authn_dbd.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
         mod_authn_dbm.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_authn_default.so:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_authn_file.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_authz_dbm.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_authz_default.so:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
         mod_authz_groupfile.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
         mod_authz_host.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
         mod_authz_owner.so:     ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_authz_user.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
         mod_autoindex.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_cern_meta.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_cgid.so:            ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_cgi.so:             ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_dav_fs.so:          ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_dav.so:             ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_dbd.so:             ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_deflate.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_dir.so:             ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_dumpio.so:          ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_expires.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_ext_filter.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_filter.so:          ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_headers.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_ident.so:           ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
     mod_imagemap.so:        ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_include.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_info.so:            ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
     mod_log_config.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_log_forensic.so:    ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
     mod_logio.so:           ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
     mod_manager.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_mime_magic.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
     mod_mime.so:            ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_negotiation.so:     ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
     mod_proxy_ajp.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_proxy_balancer.so:  ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_proxy_cluster.so:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_proxy_connect.so:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_proxy_ftp.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_proxy_http.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_proxy_scgi.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_proxy.so:           ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_reqtimeout.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
        mod_rewrite.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
       mod_setenvif.so:        ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_slotmem.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
      mod_speling.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
   mod_ssl.so:             ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
  mod_status.so:          ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
 mod_substitute.so:      ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
 mod_unique_id.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
 mod_userdir.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
 mod_usertrack.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
 mod_version.so:         ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
 mod_vhost_alias.so:     ELF 64-bit LSB shared object, x86-64, version 1    (SYSV), dynamically linked, not stripped
  • What modules are you loading and what is the output of httpd -V and ldd /path/to/httpd. You might also grab a few lines just prior to the segfault from the access logs from all these servers you are seeing this on and look for common patterns. – Aaron Jun 20 '16 at 19:19
  • I have updated the questions with the modules being used in httpd.conf file. httpd -v gives the same output as mentioned in the question. Server version: Apache/2.2.29 (Unix) Server built: Mar 4 2016 15:21:54 – user2607367 Jun 21 '16 at 9:59
  • Without a known bug for your software version, then the first thing you should be looking at is your core files, using GDB see also stackoverflow.com/questions/8305866/… (hint: use the command bt to a stack trace) – symcbean Jun 21 '16 at 10:43
  • I am not sure how to use gdb. Can you explain how to use it? I am not getting much help from the forum that you have mentioned. – user2607367 Jun 21 '16 at 11:14
  • I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy. – user2607367 Jun 21 '16 at 14:03
0

For GDB use https://httpd.apache.org/dev/debugging.html.

You would be able to trackdown the actual cause of the error.

If the crash is hard to reproduce it may be a good idea to configure Apache to only use one child processes for handling requests. The config is something like this:

StartServers 1

MinSpareServers 1

MaxSpareServers 1

Check your loaded configuration file- php.ini parameters.Configure it as per the server configuration(H/W resources)

Check if any page caching technique used like APC or memcached, if yes then allocate resources accordingly.

  • I have updated a question with a common pattern observed in the error logs when segmentation fault occurs. It seems to be occurring due a bug in mod_proxy. – user2607367 Jun 21 '16 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.