2

I'm running apache 2.2.25 with prefork and mod_perl on CentOS (mostly 6.4). Recently one instance went particularly haywire, forking off children of children of children:

<snip>
\_ /usr/sbin/httpd
|   \_ /usr/sbin/httpd
|   |   \_ /usr/sbin/httpd
|   |       \_ /usr/sbin/httpd
|   |           \_ /usr/sbin/httpd
|   |           |   \_ /usr/sbin/httpd
|   |           |       \_ /usr/sbin/httpd
|   |           |       |   \_ /usr/sbin/httpd
|   |           |       \_ /usr/sbin/httpd
|   |           \_ /usr/sbin/httpd
|   \_ /usr/sbin/httpd
|       \_ /usr/sbin/httpd
|       |   \_ /usr/sbin/httpd
|       |   |   \_ /usr/sbin/httpd
|       |   |   |   \_ /usr/sbin/httpd
|       |   |   \_ /usr/sbin/httpd
|       |   \_ /usr/sbin/httpd
|       \_ /usr/sbin/httpd
\_ /usr/sbin/httpd
|   \_ /usr/sbin/httpd
\_ /usr/sbin/httpd
\_ /usr/sbin/httpd
</snip>

At the same time the box ran out of memory (I'm not sure which came first, the deep forking or the memory running out).

I've never seen apache do this before and haven't yet narrowed down the cause. The two possibilities I'm investigating are a memory leak in mod_perl (or rather in our code which then persists because of mod_perl) or a security exploit that allows for some limited fork bomb (can't fork off arbitrary children but can fork off additional apache children).

I'm not sure what would cause apache to fork like this (edit I originally mentioned a mistaken belief about how graceful worked) but my thought here is that a circular ref type memory leak (either directly or indirectly by just using all the available memory on the box) caused some kind of mod_perl confusion which resulted in additional worker children forking from children instead of the main parent. I have no real reason for this, just attempting to reconcile the two problems (deep forking and memory usage).

Alternatively, we may have a security exploit that I need to track down.

Has anyone seen this type of behavior from apache before and know of a solution?

  • That part about a child becoming the new parent is not accurate. The parent is fixed during a graceful restart – covener Jan 14 '14 at 1:47
0

It turns out this was code forking under mod_perl (which is already a bad idea) and then making it worse by not cleaning up the child. Under mod_perl, when you fork, you're not forking your code, you are forking apache. So then that child goes on to continue serving content as any other forked apache child but doesn't get included in the count and therefore won't ever get culled if you get too many instances.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.