Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

My server does this every few days. What sucks is that it always seems to do this right after I go to bed, so when I wake up, I'm greeted with the fact that my server has been down for the past 6 or 7 hours.

When I first noticed this, I added a cronjob that tries to restart the server every 15 minutes, but I guess that didn't fix it. Once I noticed the server was down, I can this command:

/etc/init.d/apache2 restart
* Restarting web server apache2
apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
... waiting ...........................................................apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
httpd (pid 17597) already running

...which is odd, because a restart should restart the server, even if it's already running, correct? I eventually had to "stop" then "start" to get it working again.

I then looked through the logs, and found something very weird. It seems that around the time the server crashed, the logs have entries that are wildly out of order. It looks a little like this:

xx.xxx.xxx.x - - [21/Apr/2010:06:32:05 -0400] "GET / blah"
xx.xxx.xxx.x - - [21/Apr/2010:06:51:25 -0400] "GET / blah"
x.xx.xxx.xxx - - [21/Apr/2010:06:38:23 -0400] "GET / blah"
xxx.xx.xx.xx - - [21/Apr/2010:06:31:56 -0400] "GET / blah"
xxx.xx.xx.xx - - [21/Apr/2010:06:51:49 -0400] "GET / blah"
xx.xx.xxx.xx - - [21/Apr/2010:06:33:20 -0400] "GET / blah"

I don't think the problem is memory, because this:

tells me that right before the crash, memory usage is fine.

I'm running apache with the worker mpm, here are the settings for that:

<IfModule mpm_worker_module>
  StartServers            1
  MaxClients            100
  MinSpareThreads         5
  MaxSpareThreads        10
  ThreadsPerChild        10
  MaxRequestsPerChild  3000
</IfModule>

This apache server is running a bunch of stuff, but most of the traffic comes from a django project I'm hosting, that uses mod_wsgi. There also is a simple machines forum that is running off of mod_fcgid. Those setting are below:

<IfModule mod_fcgid.c>
  MaxRequestsPerProcess 500
  MaxProcessCount 3

  AddHandler fcgid-script .php .fcgi
  AddHandler cgi-script .cgi .pl
  FCGIWrapper "/usr/bin/php-cgi" .php 
</IfModule>

Anyone know of anything else I can check? I've just about tweaked every single setting I can think of, yet these freezes still happen.

Edit: I have both a postgres and mysql server running on this machine, but they both work during this freeze, because my backup script ran during that 5 hour time frame, and it worked perfectly fine.

Edit2: I'm running Ubuntu Server 9.10. When the server is down, all requests just never return. The page hangs. No error messages or anything.

share|improve this question
    
Have you looked through the other cron jobs to eliminate another process that may be killing Apache? As for restart, it should be running a stop and start anyway, so you may find it's not waiting long enough for the stop before it tries to start again? Just a thought. –  user3914 Apr 21 '10 at 17:17
    
@Randolph, you may be right with that. When I ran the restart script, it sis take a very long time. Also, there are no other cron jobs that are ran during the time that apache always freezes. The weird thing is that it always seems to want to freeze in the early morning when load is the lowest. –  nbv4 Apr 21 '10 at 17:21
    
@voretaq might be on to something with MaxClients ... –  user3914 Apr 21 '10 at 17:43
    
@Randolph if his logs supported it I'd agree, but I've never hit MaxClients without a log entry to that effect (it may be possible but it's never happened to me). setting it to 150 would determine empirically if it's being hit though: the next crash would have 150 apache processes instead of only 100. –  voretaq7 Apr 21 '10 at 17:55
2  
I think what may be happening is that when apache get a request, it hands it over to mod_wsgi for a response, but mod_wsgi doesn't return anything, so those connections hang. Over the course of an hour, over 100 of those requests persist, reaching the MaxClients, then causes the crash. –  nbv4 Apr 21 '10 at 18:22

5 Answers 5

up vote 3 down vote accepted

You don't say anything how you are using mod_wsgi and have it configured. I would suggest as a start to read 'http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API'. You possibly are using a C extension module for Python which doesn't implement full threading properly. If you use daemon mode of mod_wsgi though, such deadlocks should be detected and processes at least forcibly restarted after a period. So, if you are using embedded mode, which is discouraged, then use daemon mode instead as a start.

Overall, this sort of issue, if you believe it is related to mod_wsgi should be discussed on the mod_wsgi mailing list. Debugging stuff like this on StackOverflow/ServerFault/SuperUser is really hard.

share|improve this answer
    
I'm almost 100% sure the problem is with mod_wsgi. I've upgraded from 2.5 to 2.8 and that seems to have fixed it. –  nbv4 May 6 '10 at 5:19
    
The changes between mod_wsgi 2.5 and 2.8 were very very minor and nothing changed which could have resulted in a problem like that disappearing. Are you sure nothing else changed along the way. For example, updated Python version or updated versions of third party modules for Python or even Apache itself? –  Graham Dumpleton May 6 '10 at 6:27

Well, it appears something is causing your web server to get a metric ass-ton of requests -- If you look in your apache error log you'll probably see that you're hitting your MaxClients limit (which is why your site falls over).

Find and eliminate the source of the request storm and your problem will go away (if you're lucky it's all from one source and you can just block them at your firewall).

Alternatively you can crank MaxClients up to some insane value, but that will probably just upset the rest of your system.

share|improve this answer
    
I just looked through my error log, theres one message at 5:28, then the next error mesage is at 12:15, which is after my restart. No mention of MaxClients anywhere... –  nbv4 Apr 21 '10 at 17:47
    
Hmm, this still smells like you're hitting MaxClients (just judging by your graph: You top out at your MaxClients limit), but it's really strange that your logs don't show it happening. Do the crashes happen around the same time every night or do they move around? –  voretaq7 Apr 21 '10 at 17:51
    
Its usually in the early morning, and it's not every day. Last time it happened it was the 17th. Before that it happened twice in February. They occur anywhere between 3 AM and 8 AM. –  nbv4 Apr 21 '10 at 18:11
    
This is completely off-topic, so I hope you don't mind: I just read your comment on the Lawyers USA Online site and I thank you for showing sanity in a crazy world. "Context is important folks: It's not as dumb a question as you all think it is." I wish I could +1 you there too :-). Sorry for hijacking this thread. –  user3914 Apr 22 '10 at 5:09

I would guess it is one of the modules, or it could be some interaction between the modules. My first suspect would be mod_wsgi, especially since you are using it with MPM worker. It should be safe, according to the developers, but it still creates a python interpreter per process, and the python interpreter is not exactly thread-friendly. Try switch your django application to fastcgi. Or try run apache with MPM prefork.

Then you could try switching from mod_fcgid to mod_fastcgi, and/or try disable other modules you may have enabled.

share|improve this answer

Can you post what you have in error_log (can be found in /var/log/httpd/error_log) when the problem happens?
Also, I would like to see parts from /var/log/messages from the same time.
And, post the output of df -h (disk usage).

share|improve this answer

Your problem could be any number of things, but since it's clear you're not already the first thing you need to do is install Monit or some similar software. Monit is a daemon that runs on your server and, as long as the OS is running, makes regular checks that applications you define are running. You can tell it to check that Apache is available and if it's not restart apache. You can also tell it to restart apache depending on system variables like high load or full ram. Once you have that set up you can at least know that your site won't go down when this happens, and Monit will email you when it takes action, so you'll have an easy log of when the problem occurs to compare with logs etc.

http://mmonit.com/monit/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.