I am investigating a strange behavior on 3 CentOS machines serving as a MongoDB replica set, whereas one of them is also hosting a PHP web app utilizing the MongoDB replica set. The basic setup is as follows:
- Node 1: CentOS 5.8, MongoDB 2.6.10 (acting as PRIMARY), Apache 2.2.23 (running a PHP web app with MongoDB driver 1.6.10)
- Node 2: CentOS 5.11, MongoDB 2.6.10 (acting as SECONDARY), Apache 2.2.23 (running nothing than an empty index.html, called up by Nagios every few minutes)
- Node 3: CentOS 5.11, MongoDB 2.6.10 (acting as SECONDARY), Apache 2.2.23 (running nothing than an empty index.html, called up by Nagios every few minutes)
Now, all of them are experiencing a nearly constant 100% CPU load. The load is caused by a large number of httpd processes being launched. Even on nodes 2 and 3, which have almost no HTTP traffic. The CPU usage of the mongod process is vanishingly small on each machine.
This is what the
top output on node 2 looks like:
The output looks very similar on nodes 1 and 3. This is what the httpd access log on node 2 looks like:
Having a large number of httpd processes but a very small amount of actual HTTP requests seems strange to me.
When I check
netstat -p on node 2 I see something like this:
The open mongod sockets should be the replication workers or replica set heartbeats, but what is really striking in the
netstat -p output: The additional number of open httpd (?!) sockets coming from MongoDB port 27017 on their counterpart (node 3).
As a result of this, logging on to the machines (e. g. via SSH) becomes tremendously slow after a while. Restarting httpd works in the short term, the number of httpd processes and CPU load instantly drops to a normal level. But after a few hours, the httpd processes/sockets fill up again and the machine is back on 100% CPU load. Restarting httpd does not have any impact on the replica set's operation.
I am not sure but I guess the prefork/worker configuration of Apache is nothing special:
<IfModule prefork.c> StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 </IfModule> <IfModule worker.c> StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule>
To cut a long story short...
- Is it really the MongoDB replica set, which is somehow affecting the web server and why?
- Why does Apache (httpd) even care about the sockets buzzing on these random 37000 ~ 60999 ports? (Shouldn't it just handle ports 80/443?)
- What can I do to solve or at least to isolate the problem?