3

[EDIT]

Bad question, mostly - see my answer

[/EDIT]

We've got a system running RHEL6, x64. We are using a local installation of apache 2.2.22 from source. we serve primarily:

  1. mod_perl applications (with a local installation of perl 5.16.0)
  2. tomcat applications proxied with mod_jk

Here is some context; the main question is below.

All of this talks to an Oracle backend.

We are having issues with Oracle becoming unresponsive. We think this is because we're hitting the maximum process limit in oracle. We've upped the process limit, but now we are hitting memory pressure on the oracle server. We have tons of oracle sessions sitting idle. I can trace a bunch of them back to the httpd processes.

We have mod_perl's Apache::DBI start up a new connection to the database with each httpd child that's spawned. We are concerned that these are not always getting closed out properly when the httpd's exit...and the httpd's are exiting very frequently. I know that it would be good to modify the mod_perl applications to use some better form of db connection pooling; we plan to pursue that but would like to solve our immediate problem sooner.

So here's the main question.

We are using the prefork MPM.

The apache child processes are lasting at most a few minutes. Log analysis shows that each one is serving fewer than 50 clients before exiting; the last request each child serves is OPTIONS * HTTP/1.0 on some sort of internal connection; I'm under the impression that this is a "ping" from the master process.

I've adjusted the MPM config as follows. I didn't want to raise MinSpareServers too high, because, after all, i'm trying to minimize the number of sessions to oracle.

MinSpareServers       5
MaxSpareServers       30
MaxClients            150
MaxRequestsPerChild   10000

Right now we're serving 250-300 requests per minute.

We've got 21 httpd's running, the eldest (other than the master, owned by root) being 3 minutes old.

This rate of reaping of the apache children really seems excessive. What could be causing it?

Apache was built with:

  $ ./configure --prefix=/opt/apache --with-ssl=/usr/lib --enable-expires --enable-ext-filter --enable-info --enable-mime-magic --enable-rewrite --enable-so --enable-speling --enable-ssl --enable-usertrack --enable-proxy --enable-headers --enable-log-forensic

Apache config info:

% /opt/apache/bin/httpd -V
Server version: Apache/2.2.22 (Unix)
Server built:   Jul 23 2012 22:30:13
Server's Module Magic Number: 20051115:30
Server loaded:  APR 1.4.5, APR-Util 1.4.1
Compiled using: APR 1.4.5, APR-Util 1.4.1
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/opt/apache"
 -D SUEXEC_BIN="/opt/apache/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

modules are compiled into apache rather than shared libs:

% /opt/apache/bin/httpd -l
Compiled in modules:
  core.c
  mod_authn_file.c
  mod_authn_default.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_default.c
  mod_auth_basic.c
  mod_ext_filter.c
  mod_include.c
  mod_filter.c
  mod_log_config.c
  mod_log_forensic.c
  mod_env.c
  mod_mime_magic.c
  mod_expires.c
  mod_headers.c
  mod_usertrack.c
  mod_setenvif.c
  mod_version.c
  mod_proxy.c
  mod_proxy_connect.c
  mod_proxy_ftp.c
  mod_proxy_http.c
  mod_proxy_scgi.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  mod_ssl.c
  prefork.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_info.c
  mod_cgi.c
  mod_negotiation.c
  mod_dir.c
  mod_actions.c
  mod_speling.c
  mod_userdir.c
  mod_alias.c
  mod_rewrite.c
  mod_so.c

One final note - the red hat httpd, apr, and perl packages are all installed, but ldd shows that none of those libraries are linked with the running httpd.

6
  • By 'Oracle backend' you mean database? Is the code using persistent connections; and is a connection pool in place if so?
    – thinice
    Sep 6, 2012 at 17:12
  • Yes, i mean an oracle database. Some of our legacy code is not using a connection pool, it's opening a new db connection each time an httpd process starts. Fixing that is on the list but understanding the prefork behavior seems more short-term attainable.
    – Dan Pritts
    Sep 6, 2012 at 18:44
  • Outside of setting the Apache KeepAlive directive; prefork is all about rapid connection spawning and teardown.
    – thinice
    Sep 6, 2012 at 20:17
  • prefork is all about keeping children alive as long as practical, to avoid the high cost of starting processes. The prefork documentation even suggests setting MaxRequestsPerChild to 0, which is infinite. I've run plenty of servers before and this is unusual behavior.
    – Dan Pritts
    Sep 7, 2012 at 0:14
  • MaxRequestsPerChild 0 is a good way to hold onto memory leaks; I've always felt disagreement with that suggestion and have seen poorly run PHP sites bring prefork to it's knees with that setting. A high value, sure; but it should clean up after itself after a while. With java using persistent connections as described, I'd say that's it. At 5RPS (300/min = 5Reqs/S) - prefork is likely not the culprit but improper DB connection handling via tomcat/Java.
    – thinice
    Sep 7, 2012 at 2:03

1 Answer 1

0

aaargh

the answer in this case turns out to be that i hadn't configured apache with those maxrequestsperchild, minspareservers, and maxspareservers options.

I modified a config file that I believed was Included in the primary file, but wasn't. New to this site, and crufty old config. sigh.

So, the behavior I describe is the compiled-in default (i triple checked; it is not getting set anywhere else). The documentation suggests that these are the compiled-in defaults:

http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild

http://httpd.apache.org/docs/2.2/mod/prefork.html

MaxRequestsPerChild 10000
MaxSpareServers 10
MinSpareServers 5

I think the behavior I saw was not consistent with these defaults. I'll file a bug with apache.

I guess this is still a useful question to leave on the site, because the default behavior doesn't match the documentation. If others disagree I'll gladly close my question.

1
  • I went ahead and installed those specific config entries (matching the defaults) to see what happened. I saw the same behavior. It is really counterintuitive. I guess it really depends on how frequently the master httpd checks the traffic level and reaps its children.
    – Dan Pritts
    Sep 10, 2012 at 2:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .