Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have apache2 running on a server.
Today there were a lot of http requests (but not unusual) and somehow the load average went up to over 200(!!!). Obviously the website was down (not reachable).

enter image description here

Stopping apache would lead to a relatively quick drop in load but as soon as I started it again the load went over 100 within seconds.
What's weird is that the CPU and MEM workload were normal or even low as if the system did not "realize" there were a lot of processes to handle.

What's even weirder is that all of the sudden the CPU load went to 100% on all cores: enter image description here

From there on the load went back to normal (<1) within minutes and the website was suddenly reachable again.

I really cannot explain myself this kind of behaviour. Can anyone help me prevent it in the future?

share|improve this question
5  
You might want to start by checking Apache's logs to find out what it was doing. –  David Schwartz Jan 9 '13 at 12:19
    
You should check your webserver logs in the first place. Make sure that you're using recent version of Apache (there were some DoS vulnerabilities in Apache that recently got fixed). –  FINESEC Jan 9 '13 at 16:42

4 Answers 4

Two wild guesses:

1) Document root is served over NFS or some other network file system, or a cluster file system, which is responding slowly or not at all.

2) Your Apache (and PHP scripts or whatever) are waiting for database or some other external resource.

My first guess would be 1) since everything returned to normal in a very short spike. If that's the case, check your network or the file server.

share|improve this answer
    
Your first guess could be right since we just added a NFS. In the error.log I found some interesting information (maybe that helps): [warn] child process 2373 still did not exit, sending a SIGTERM (there were plenty of those). And then after restarting the web server: [error] server reached MaxClients setting, consider raising the MaxClients setting –  horen Jan 10 '13 at 9:28
    
NFSv4 can still occasionally surprise you especially with older kernels, and not in a positive way. NFSv3 should perform quite OK. Just make sure you tune its [rw]size mount options to suit your network, benchmark to find out what works for you. Usually 32768 or 65536 is a good value. –  Janne Pikkarainen Jan 10 '13 at 10:26

In complement to @Janne Pikkarainen first guess,

Also check in the error log messages, especially the default one if you have several log files on virtualhosts, about (internal dummy connection) signatures. details here

On some apache2 versions theses signal the apache internal connections set to enforce gracefull reloads or garbage collections. They are send to all alive children and managed by your default virtualhost (the first one from apache2 -S). If your default virtualhost is resource intensive this sudden peak could produce such bad effects also, not only log garbage.

if your NFS or database -- or any other blocking thing -- is hit by theses an easy fix is to have a real dummy default virtualhost (the «it works» one).

share|improve this answer

The prefork MPM is ridiculously unsuited to a high-performance production deployment. Starting up extra processes is very expensive, and each process(thread) can take up to 30MB easily.

Instead, consider using the worker MPM or (when running a modern apache) the Event MPM.

One can trivially increase the threadcount by a factor of 10 with the same memory footprint.

share|improve this answer
    
Is it possible to use worker/event with PHP? I thought it was a no-go. –  pauska Jan 10 '13 at 14:49
    
As long as you don't use php extensions that cause trouble with multithreading, then yes, it is quite possible. Alternatively, use FCGI instead; you will get an additional performance boost on PHP too. –  adaptr Jan 10 '13 at 14:51
    
yeah FastCGI is really the way to go these days.. –  pauska Jan 10 '13 at 14:56

Take a look at your Apache config, and pay attention to the following directives:

The default configuration for those might not be appropriate. This was the case for me, and resulted in Apache using lots of memory when the server got hit more than usual. My issue was resolved by reducing the values of these.

Make sure that you understand the configuration of your Production Apache - defaults are there for testing.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.