3

Our Apache+Django server has the problem that workers get stuck. It's an mpm worker model, and after some time, each process that serves some dozen worker threads has all its workers frozen:

# apache2ctl status
Apache Server Status for localhost

Server Version: Apache/2.2.14 (Ubuntu) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_wsgi/
    2.8 Python/2.6.5
Server Built: Mar 8 2013 16:46:38

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Current Time: Friday, 05-Apr-2013 15:56:17 CEST
Restart Time: Thursday, 04-Apr-2013 11:23:23 CEST
Parent Server Generation: 11
Server uptime: 1 day 4 hours 32 minutes 53 seconds
Total accesses: 244313 - Total Traffic: 4.7 GB
CPU Usage: u181.45 s33.97 cu.62 cs0 - .21% CPU load
2.38 requests/sec - 47.9 kB/second - 20.2 kB/request
108 requests currently being processed, 42 idle workers

_K__K______KK_____W_________W________K_K__________..............
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW..............
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW..............
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

When doing apache2ctl fullstatus, you can see that it's exactly two PIDs that have all their workers in "working" state. Currently, PID 822 and 5284. And, those processes aren't serving any functional requests. Additionally, they can only be killed with signal 9 (kill -9)

The option WSGIDaemonProcess cpu-time-limit=120/120 is not going to help us for two reasons: Only WSGI version 3.0 and higher have it, plus, the processes aren't consuming CPU, so their CPU time is low.

We experience some slowness with the server. It's not super slow, but it can be faster (sometimes it hangs on requests) and I suspect this problem is related. In any case, it's not supposed to be like this.

It's an Ubuntu 10.04 LTS server with Apache 2.2.14 and libapache2-mod-wsgi 2.8-2ubuntu1. Sites are served like:

WSGIScriptAlias / /srv/http/bla/passenger_wsgi.py

This is the worker config:

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      50
    MaxClients           200
    ServerLimit          6
    MaxRequestsPerChild  1000
</IfModule>

Any idea what this is and how to solve it? Or, at least how to set some automatic kill on those processes? Ulimit is hard, because they don't consume CPU a lot.

2 Answers 2

2

You MPM settings are a bit broken for various reasons for a start. Suggest you watch my talk from PyCon at:

As to your server hanging, you likely have a third party extension module being used which isn't safe to use from a sub interpreter. You need to force your application to run in the main interpreter. See:

For working out where processes are hanging also see means of getting stack traces as described in:

If it is a deadlock as expect and don't want to just try using the main interpreter, likely will need to use gdb to get stack traces of where it is stuck.

The Python stack traces approach will work if the problem is that your code is blocking on calls to external services. You can also perhaps get an idea of that by looking at open file descriptors using lsof or ofiles.

0

I fixed it some time ago by converting the sites to use daemon mode, instead of embedded mode, and putting an nginx proxy in front of it, which handles all the static file serving.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .