Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I requested an hard reboot because none of ssh and http worked. Ping worked normally.

Which logs should i check to understand what was the problem?

Thanks! (debian 6 on lamp)

Edit: my memory and swap:

Mem:   4040068k total,  1114920k used,  2925148k free,   109212k buffers
Swap:  1051384k total,        0k used,  1051384k free,   283820k cached

4 GB ram

(and more than 1TB of HDD)

The cause is from 2 days ago:

look how the usage of swap goes +60% in less than 10hours

My control panel reports this as top 5 memory usage process:

If every apache2 process is 190MB large that sux because IF i do TOP i have 262 sleeping process most of them are apache2!

My apache mpm_prefork settings are:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    ServerLimit      1500
    MaxClients            1500
    MaxRequestsPerChild   2000
</IfModule>

KeepAlive On


MaxKeepAliveRequests 100


KeepAliveTimeout 4
share|improve this question
2  
Change your apache configuration to not start more processes than you have RAM for. –  mattdm Mar 8 '11 at 15:41
    
@mattdm: i posted my apache2 config, any suggestions? Maybe i should lower MaxRequestsPerChild to avoid apache2 memory leaks? –  dynamic Mar 8 '11 at 15:56
    
Do the math here. If you have 1500 servers, even if they're a svelte 10MB each, that's 15GB of RAM. If 190MB is a typical size for an apache process with your workload (php?), and you have 4GB of ram, ServerLimit should be no more than 21. –  mattdm Mar 8 '11 at 16:17
2  
Also, I suspect that you don't have a memory leak, but rather a PHP script which is able to consume a lot of RAM. Apache processes will expand to the maximum memory used by PHP, and not go back down. So setting MaxRequestsPerChild ridiculously low will mitigate this, but the solution is to either a) reign in your PHP app's memory consumption or b) switch to a different server architecture (using fastcgi for php). –  mattdm Mar 8 '11 at 16:22
    
@mattdm: Aren't resources freed by PHP when the scripts are done? –  dynamic Mar 8 '11 at 16:27
show 3 more comments

1 Answer 1

up vote 4 down vote accepted

Which logs should i check to understand what was the problem?

All of them. ping working just means enough of the IP stack is up to process ICMP Echo requests (that's not a huge portion of the system compared to what's required for SSH and web servers). You could have had what I call a "partial panic" (Kernel blew up, but the IP code kept running), run out of RAM, or your SSH/HTTPd processes could have fallen over for unspecified reasons.

/var/log/messages is probably a good starting point, as is the log for your web server (presumably Apache). If nothing else it will give you an idea of when the system last worked and how long it was in the brain-dead state before it got rebooted...


Update based on comment

Sounds like something has a memory leak.
When you ran out of swap userland blew up but the kernel (being wired in RAM) could keep running & answering ping requests.

For a permanent resolution you should monitor your swap utilization carefully and when you notice it trending dangerously upward (>33% used is my threshold) hunt down the process with the most swap used: That's probably your culprit.

share|improve this answer
    
thanks for the reply, in that log i found Mar 8 15:40:20 ns354729 kernel: Free swap = 0kB before the down I could see (in control panel of my hosting company) the values of CPU, RAM and SWAP where totally out of control with 5% of free ram and only 1-2% of SWAP. –  dynamic Mar 8 '11 at 15:20
    
But now the problem is: How can I know what caused all this problem –  dynamic Mar 8 '11 at 15:23
1  
@yes123 - See the update to the answer :-) You could also have an undersized box (too little RAM/too little swap for what you're trying to do), but a memory leak is what I'd investigate first. –  voretaq7 Mar 8 '11 at 15:25
    
@voretaq: i added some information. As you can see my swap grew in few hours by +70% –  dynamic Mar 8 '11 at 15:34
    
@yes123 that's definitely a memory leak (or a bunch of processes being kicked off and chewing through your swap). To find the cause you have to watch it happening. Also note that "top memory users" can be a somewhat misleading measurement (you want "top swappers" -- If you run top it's the guys with the biggest SWAP column -- reorder top's output to see that). –  voretaq7 Mar 8 '11 at 15:41
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.