Our Apache+Django server has the problem that workers get stuck. It's an mpm worker model, and after some time, each process that serves some dozen worker threads has all its workers frozen:
# apache2ctl status
Apache Server Status for localhost
Server Version: Apache/2.2.14 (Ubuntu) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_wsgi/
2.8 Python/2.6.5
Server Built: Mar 8 2013 16:46:38
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Current Time: Friday, 05-Apr-2013 15:56:17 CEST
Restart Time: Thursday, 04-Apr-2013 11:23:23 CEST
Parent Server Generation: 11
Server uptime: 1 day 4 hours 32 minutes 53 seconds
Total accesses: 244313 - Total Traffic: 4.7 GB
CPU Usage: u181.45 s33.97 cu.62 cs0 - .21% CPU load
2.38 requests/sec - 47.9 kB/second - 20.2 kB/request
108 requests currently being processed, 42 idle workers
_K__K______KK_____W_________W________K_K__________..............
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW..............
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW..............
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
When doing apache2ctl fullstatus
, you can see that it's exactly two PIDs that have all their workers in "working" state. Currently, PID 822 and 5284. And, those processes aren't serving any functional requests. Additionally, they can only be killed with signal 9 (kill -9
)
The option WSGIDaemonProcess cpu-time-limit=120/120
is not going to help us for two reasons: Only WSGI version 3.0 and higher have it, plus, the processes aren't consuming CPU, so their CPU time is low.
We experience some slowness with the server. It's not super slow, but it can be faster (sometimes it hangs on requests) and I suspect this problem is related. In any case, it's not supposed to be like this.
It's an Ubuntu 10.04 LTS server with Apache 2.2.14 and libapache2-mod-wsgi 2.8-2ubuntu1. Sites are served like:
WSGIScriptAlias / /srv/http/bla/passenger_wsgi.py
This is the worker config:
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 50
MaxClients 200
ServerLimit 6
MaxRequestsPerChild 1000
</IfModule>
Any idea what this is and how to solve it? Or, at least how to set some automatic kill on those processes? Ulimit is hard, because they don't consume CPU a lot.