2

I try to find the cause why my server gets out of memory every few weeks and calls the OOM killer. It seems to be that memory usage is stable for about 2 weeks, then going up gradually for 2 weeks. Then there is a big hike resulting in an OOM call.

http://i.stack.imgur.com/PNIO5.jpg

The memory usage just before the hike:

Wed Jun  3 08:50:01 EDT 2015                                                                                                                   
COMMAND         %MEM                                                                                                                           
tor             14.7
mysqld          11.8                                                                                                                           
spamd child      5.5                                                                                                                           
apache2          4.6                                                                                                                           
spamd child      4.5                                                                                                                           
apache2          4.4                                                                                                                           
apache2          4.4                                                                                                                           
apache2          4.2                                                                                                                           
apache2          4.2                                                                                                                           
apache2          3.3                                                                                                                           
Memory Space Details                                                                                                                           
Total Memory space : 1250 MB                                                                                                                   
Used Memory Space : 964 MB                                                                                                                     
Free Memory : 285 MB                                                                                                                           
Swap memory Details                                                                                                                            
Total Swap space : 255 MB                                                                                                                      
Used Swap Space : 81 MB                                                                                                                        
Free Swap : 174 MB 

Here is the memory usage just before OOM is called.

Wed Jun  3 09:10:02 EDT 2015                                                                                                                   
COMMAND         %MEM                                                                                                                           
tor             13.4
mysqld          11.2                                                                                                                           
spamd child      5.2                                                                                                                           
apache2          3.8                                                                                                                           
apache2          3.6                                                                                                                           
apache2          3.3                                                                                                                           
apache2          2.6                                                                                                                           
apache2          2.5                                                                                                                           
apache2          2.5                                                                                                                           
spamd child      2.4                                                                                                                           
Memory Space Details                                                                                                                           
Total Memory space : 1250 MB                                                                                                                   
Used Memory Space : 873 MB                                                                                                                     
Free Memory : 376 MB                                                                                                                           
Swap memory Details                                                                                                                            
Total Swap space : 255 MB                                                                                                                      
Used Swap Space : 178 MB                                                                                                                       
Free Swap : 77 MB  

System calling OOM:

1 Time(s): /usr/sbin/spamd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

The process TOR is killed.

After a reboot, memory usage drops and system is running well for a few weeks.

OMMAND         %MEM                                                                                                                           
tor             12.0                                                                                                                           
mysqld           7.8                                                                                                                           
/usr/sbin/spamd  6.2                                                                                                                           
spamd child      6.1                                                                                                                           
spamd child      6.1                                                                                                                           
apache2          4.4                                                                                                                           
apache2          2.9                                                                                                                           
apache2          2.9                                                                                                                           
apache2          2.5                                                                                                                           
apache2          2.0                                                                                                                           
Memory Space Details                                                                                                                           
Total Memory space : 1250 MB                                                                                                                   
Used Memory Space : 659 MB                                                                                                                     
Free Memory : 590 MB                                                                                                                           
Swap memory Details                                                                                                                            
Total Swap space : 255 MB                                                                                                                      
Used Swap Space : 0 MB                                                                                                                         
Free Swap : 255 MB 

Kernel output: http://pastebin.com/rBvbFcyt

Where is the memory jump coming from? Why is the memory usage gradually increasing over a period of 2 weeks? Memory leak?

  • We had an issue with mysql and a specific kernel version which caused it to go nuts after a couple of weeks. Do you have any other metrics that would show utilization over time? – Satanicpuppy Jun 5 '15 at 15:11
  • I run munin...what are you looking for? I also run a script, writing memory usage to a file. mysql is stable at 11% – MikeSkril Jun 5 '15 at 15:20
1

Hard to say what's causing the jump or the gradual increase with the info provided.

The snapshots provided suggests that the top memory users are actually behaving quite OK - their memory footprint is lower during the memory crunch, right before the OOM killer strikes. Which suggests some OTHER processes actually contribute to the crunch (maybe related to some user activity, maybe some cron stuff, maybe some transient condition suddenly lasting much longer than usual?).

Unfortunately the OOM killer is not smart enough and blindly kills the top memory user process - IMHO a rather uninspired decision on servers with dedicated functions.

A much more detailed monitoring would be necessary to uncover the processes causing the memory crunch. I'd start with periodic top executions dumping info about all the processes to a file which can be examined later. Why all the processes? Because a sudden spike or a gradual increase in the number of processes (even with a small memory footprint each) can also potentially explain memory usage changes like the ones you reported.

  • I now run a script to write memory statistics to a file every 5 minutes. I did that so far for the top 10, but extended now to all processes. – MikeSkril Jun 5 '15 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.