Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am having a problem with an OSQA site. It is Django/Apache/mod_wsgi configured site. Every hour, the CPU spikes to 164% (Average) for task HTTPD. After 10 minutes, it frees back up. I have reviewed the logs, cron tables, made many config changes, but cannot track this problem down. Can someone please look at the information below and let me know if it is a configuration problem, or if anyone else has experienced this issue.

  1. Running TOP shows HTTPD using 165% of CPU
  2. VMware performance monitor also displays spikes.
  3. This happens every hour for 10 minutes.
  4. I have the following information from server status

    Server Version: Apache/2.2.15 (Unix) DAV/2 mod_wsgi/3.2 Python/2.6.6
    Server Built: Feb 7 2012 09:50:15
    
    Current Time: Sunday, 10-Jun-2012 21:44:29 EDT
    Restart Time: Sunday, 10-Jun-2012 19:44:51 EDT
    Parent Server Generation: 0
    Server uptime: 1 hour 59 minutes 37 seconds
    Total accesses: 1088 - Total Traffic: 11.5 MB
    CPU Usage: u80.26 s243.8 cu0 cs0 - 4.52% CPU load
    .152 requests/sec - 1682 B/second - 10.8 kB/request
    4 requests currently being processed, 11 idle workers
    
    ....._..........__......W.......................................
    ...................................C._..._....._L__._L_._.......
    ......................
    
    Scoreboard Key:
    "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
    "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
    "C" Closing connection, "L" Logging, "G" Gracefully finishing,
    "I" Idle cleanup of worker, "." Open slot with no current process
    
    Srv PID Acc M   CPU     SS  Req Conn    Child   Slot    Client  VHost   Request
    0-0 -   0/0/34  .   0.42    327 17  0.0 0.00    0.67    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    1-0 -   0/0/22  .   0.31    339 32  0.0 0.00    0.26    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    2-0 -   0/0/22  .   0.65    358 10  0.0 0.00    0.31    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    3-0 -   0/0/31  .   1.03    378 31  0.0 0.00    0.60    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    4-0 -   0/0/20  .   0.45    356 9   0.0 0.00    0.31    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    5-0 18852   0/16/34 _   0.98    27  18120   0.0 0.37    0.62    69.180.250.36   osqa.informs.org    GET /questions/289/what-is-the-difference-between-operations-re
    6-0 -   0/0/32  .   0.94    309 29  0.0 0.00    0.64    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    7-0 -   0/0/31  .   1.15    382 32  0.0 0.00    0.75    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    8-0 -   0/0/21  .   0.28    403 19  0.0 0.00    0.20    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    9-0 -   0/0/32  .   1.37    288 16  0.0 0.00    0.60    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    10-0    -   0/0/33  .   1.72    383 16  0.0 0.00    0.40    127.0.0.1   osqa.informs.org    OPTIONS * HTTP/1.0
    
  5. I am running Django 1.3

  6. This is a mod_wsgi configuration and copied is the wsgi.conf file:

    <IfModule !python_module>
    <IfModule !wsgi_module>
    
    LoadModule wsgi_module modules/mod_wsgi.so
    
    <IfModule wsgi_module>
    
    <Directory /var/www/osqa>
     Order allow,deny
     Allow from all
     #Deny from all
    </Directory>
    
    WSGISocketPrefix /var/run/wsgi
    WSGIPythonEggs /var/tmp
    WSGIDaemonProcess OSQA maximum-requests=10000
    WSGIProcessGroup OSQA
    
    Alias /admin_media/ /usr/lib/python2.6/site-packages/Django-1.2.5-py2.6.egg/django/contrib/admin/media/
    Alias /m/ /var/www/osqa/forum/skins/
    Alias /upfiles/ /var/www/osqa/forum/upfiles/
    
    <Directory /var/www/osqa/forum/skins>
     Order allow,deny
     Allow from all
    </Directory>
    
    WSGIScriptAlias / /var/www/osqa/osqa.wsgi
    
    </IfModule>
    </IfModule>
    </IfModule>
    
  7. This is the httpd.conf file

    Timeout 120
    KeepAlive Off
    MaxKeepAliveRequests 100
    MaxKeepAliveRequests 400
    KeepAliveTimeout 3
    
    <IfModule prefork.c>
    Startservers      15
    MinSpareServers   10
    MaxSpareServers   20
    ServerLimit      50
    MaxClients       50
    MaxRequestsPerChild  0
    </IfModule>
    
    
    <IfModule worker.c>
    StartServers         4
    MaxClients         150
    MinSpareThreads      25
    MaxSpareThreads     75
    ThreadsPerChild      25
    MaxRequestsPerChild  0
    </IfModule>
    
  8. We are using MySQL

  9. The server is an ESX4i, configured for the VM to use 4 CPUs and 8 GB Ram. Hyper threading is enabled, 2 physical CPU's, with 4 Logical. the CPU are Intel Xeon 2.8 GHz. Total memory is 12GB
share|improve this question
    
The server is an ESX4i, configured for the VM to use 4 CPUs and 8 GB Ram. Hyper threading is enabled, 2 physical CPU's, with 4 Logical. the CPU are Intel Xeon 2.8 GHz. Total memory is 12GB –  Red Rover Jun 11 '12 at 2:21
    
Use the display-name option to WSGIDaemonProcess to name the mod_wsgi process and confirm whether load is from the mod_wsgi processes or actually Apache using 'ps'. Also, perhaps look at New Relic application performance monitoring and service monitoring to get more information. –  Graham Dumpleton Jun 11 '12 at 4:27
    
Um, so what's the actual problem? Does something break when this happens? –  mgorven Jun 11 '12 at 4:32
    
Thanks Graham, I will try those options. Mgroven, yes - OSQA stops responding, whereas Apache as shown above, slows down, yet does respond to server-status. –  Red Rover Jun 11 '12 at 11:43
    
I want to add the following lines for New Relic monitoring, but which file do I add it to? The osqa.wsgi or the wsgi.py file? import newrelic.agent newrelic.agent.initialize('/usr/bin/newrelic.ini') application = config.make_wsgi_app() return newrelic.agent.wsgi_application()(application) –  Red Rover Jun 11 '12 at 20:32

1 Answer 1

This problem was resolved. At the request of graham-dumpleton I added the server monitoring New Relic which pointed to a bad request from Yahoo Pipes. After contacting Yahoo, they provided the following text to block Yahoo Pipes from the website.

Configure your web server to block the user agent "Yahoo Pipes". For example, to block Pipes in Apache, add this to your virtual host block in httpd.conf:

SetEnvIfNoCase User-Agent "Yahoo Pipes" noPipes

Limit GET POST
Order Allow,Deny
Allow from all
Deny from env=noPipes
/Limit

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.