1

I've got a server running several cherrypy apps on apache2 under mod_wsgi. We're seeing constantly fluctuating load average on a box that is not serving many requests. As far as I can tell, the box is under no real CPU load, has plenty of memory, there is very little network traffic and no disk I/O occurring. We are running 13 mod_wsgi daemon processes with 5 threads per process serving 5 different applications. These are very lightweight backend service applications that don't do much processing at all. I've checked just about everything I can think of as a cause of the load flapping and was wondering if anyone here has had experience with a similar problem. Any comments greatly appreciated.

Here's a trace of load averages over the course of about 5 minutes on a staging box serving 10s of requests per minute:

~ $ sar -q 5
Linux 2.6.32-305-ec2    01/27/2011  _i686_  (1 CPU)

04:18:37 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
04:18:42 AM         0       257      1.52      1.90      1.89
04:18:47 AM         0       257      1.40      1.87      1.88
04:18:52 AM         0       257      1.28      1.84      1.87
04:18:57 AM         0       257      1.18      1.81      1.86
04:19:02 AM         0       257      1.17      1.79      1.85
04:19:07 AM         0       257      1.15      1.78      1.85
04:19:12 AM         0       257      1.14      1.77      1.84
04:19:17 AM         0       257      1.05      1.74      1.83
04:19:22 AM         0       257      0.96      1.71      1.82
04:19:27 AM         0       257      0.89      1.68      1.81
04:19:32 AM         0       256      0.82      1.65      1.80
04:19:37 AM         0       256      0.75      1.62      1.79
04:19:42 AM         0       256      0.69      1.60      1.78
04:19:47 AM         0       256      0.95      1.64      1.79
04:19:52 AM         0       256      1.20      1.67      1.81
04:19:57 AM         0       256      1.42      1.71      1.82
04:20:02 AM         0       256      1.31      1.68      1.81
04:20:07 AM         0       256      2.00      1.82      1.85
04:20:12 AM         0       256      2.64      1.96      1.89
04:20:17 AM         0       256      3.23      2.09      1.94
04:20:22 AM         0       256      2.97      2.06      1.93
04:20:27 AM         0       256      2.74      2.02      1.92
04:20:32 AM         0       256      2.52      1.99      1.91
04:20:37 AM         0       256      2.31      1.95      1.90
04:20:42 AM         0       256      2.13      1.92      1.89
04:20:47 AM         0       256      1.96      1.89      1.88
04:20:52 AM         0       256      1.80      1.86      1.87
04:20:57 AM         0       256      1.66      1.83      1.85
04:21:02 AM         0       256      1.52      1.80      1.84
04:21:07 AM         0       256      1.40      1.77      1.83
04:21:12 AM         0       256      1.29      1.74      1.82
04:21:17 AM         0       256      1.19      1.71      1.81
04:21:22 AM         0       256      1.09      1.68      1.80
04:21:27 AM         0       256      1.00      1.65      1.79
04:21:32 AM         0       256      0.92      1.62      1.78
04:21:37 AM         0       256      0.85      1.59      1.77
04:21:42 AM         0       256      0.78      1.57      1.77
04:21:47 AM         0       256      0.72      1.54      1.76
04:21:52 AM         0       256      0.98      1.58      1.77
04:21:57 AM         0       256      1.22      1.62      1.78
04:22:02 AM         0       256      1.44      1.66      1.79
04:22:07 AM         0       256      2.13      1.80      1.83
04:22:12 AM         0       256      2.76      1.93      1.88
04:22:17 AM         0       256      3.34      2.07      1.92
04:22:22 AM         0       256      3.87      2.20      1.96
04:22:27 AM         0       256      3.56      2.16      1.95
04:22:32 AM         0       256      3.28      2.13      1.94
04:22:37 AM         0       256      3.01      2.09      1.93
04:22:42 AM         0       256      2.77      2.06      1.92
04:22:47 AM         0       256      2.55      2.02      1.91
04:22:52 AM         0       256      2.34      1.99      1.90
04:22:57 AM         0       256      2.16      1.95      1.89
04:23:02 AM         0       256      1.98      1.92      1.88
04:23:07 AM         0       256      1.82      1.89      1.87
04:23:12 AM         0       256      1.68      1.86      1.86

and a top profile:

top - 04:38:57 up  1:17,  1 user,  load average: 2.55, 3.03, 2.46
Tasks:  78 total,   1 running,  77 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1741016k total,   946844k used,   794172k free,    63712k buffers
Swap:   917496k total,        0k used,   917496k free,   646064k cached

per request, apache conf of one service (they all pretty much look like this).

Listen 12800
<VirtualHost *:12800>
    WSGIScriptAlias / /var/www/services/tracking/tracking.wsgi
    WSGIDaemonProcess tracking user=www-data group=www-data processes=3 threads=5 maximum-requests=1000 umask=0007
    WSGIProcessGroup tracking
    WSGIApplicationGroup tracking
    WSGIPassAuthorization On

    ErrorLog /var/log/apache2/tracking.error.log
    CustomLog /var/log/apache2/tracking.access.log combined
    LogLevel warn
</VirtualHost> 

We haven't really done any specific parameter tuning for mod_wsgi beyond what you see in this conf.

1 Answer 1

1

Are you absolutely sure they are running in daemon mode and not in embedded mode by mistake? Embedded mode can result in undesirable behaviour for fat Python web applications depending on MPM used and MPM settings. See:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

You can also cause problems for yourself if using daemon mode and have set maximum-requests to a rediculously low value. The inactivity-timeout can also cause problems if have infrequent traffic and are using lots of processes.

Suggest you post the parts of the Apache configuration related to mod_wsgi.

3
  • Activity timeout might have something to it. We haven't configured it so it is still set to the whatever the default is. Our max requests is probably low, but it's not even approaching 1000 requests in the staging environment where I've been running experiments. Does mod_wsgi log something when the activity timeout fires or the request limit is reached? Jan 27, 2011 at 17:33
  • There is no default for inactivity timeout. Ie., not enabled if you haven't set it. You can get better logging by setting 'LogLevel' directive in Apache to 'info' instead of default of 'warn'. This will trigger mod_wsgi logging stuff about process starts/stops etc for daemon processes. Check both main Apache error log and virtual logs the daemon process group is defined in. Jan 27, 2011 at 20:55
  • 1
    BTW, if you really wanted to delve deep. You could get gdb to dump out stack traces for threads in daemon process so you can validate what they are doing. See scripted gdb part at end of section 'code.google.com/p/modwsgi/wiki/…'. You also might use 'code.google.com/p/modwsgi/wiki/…' to try and work out what is occurring at Python code level. Jan 27, 2011 at 20:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .