Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running Debian GNU/Linux 5.0 and I'm experiencing intermittent out_of_memory errors coming from the kernel. The server stops responding to all but pings, and I have to reboot the server.

# uname -a
Linux xxx 2.6.18-164.9.1.el5xen #1 SMP Tue Dec 15 21:31:37 EST 2009 x86_64
GNU/Linux

This seems to be the important bit from /var/log/messages

Dec 28 20:16:25 slarti kernel: Call Trace:
Dec 28 20:16:25 slarti kernel: [<ffffffff802bedff>] out_of_memory+0x8b/0x203
Dec 28 20:16:25 slarti kernel: [<ffffffff8020f825>] __alloc_pages+0x245/0x2ce
Dec 28 20:16:25 slarti kernel: [<ffffffff8021377f>] __do_page_cache_readahead+0xc6/0x1ab
Dec 28 20:16:25 slarti kernel: [<ffffffff80214015>] filemap_nopage+0x14c/0x360
Dec 28 20:16:25 slarti kernel: [<ffffffff80208ebc>] __handle_mm_fault+0x443/0x1337
Dec 28 20:16:25 slarti kernel: [<ffffffff8026766a>] do_page_fault+0xf7b/0x12e0
Dec 28 20:16:25 slarti kernel: [<ffffffff8026ef17>] monotonic_clock+0x35/0x7b
Dec 28 20:16:25 slarti kernel: [<ffffffff80262da3>] thread_return+0x6c/0x113
Dec 28 20:16:25 slarti kernel: [<ffffffff8021afef>] remove_vma+0x4c/0x53
Dec 28 20:16:25 slarti kernel: [<ffffffff80264901>] _spin_lock_irqsave+0x9/0x14
Dec 28 20:16:25 slarti kernel: [<ffffffff8026082b>] error_exit+0x0/0x6e

Full snippet here: http://pastebin.com/a7eWf7VZ

I thought that perhaps the server was actually running out of memory (it has 1GB physical memory), but my Cacti memory graph looks OK to me...

A friend corrected me here; he noted that the graph is actually inverted, since the purple indicates memory free (not memory used as the title suggests).

Memory usage graph

But strangely the load graph goes through the roof shortly before the kernel crashes:

Average load graph

What logs can I look at for more info?

Update:

Maybe noteworthy - the CPU percentage and network traffic graphs were both normal at the time of the crash. The only abnormality was the average load graph.

Update 2:

I think this started happening when I deployed Passenger/Ruby, and using top I see that Ruby is using most of the memory, and a fair amount of CPU:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 5189 www-data  18   0  255m 124m 3388 S    0 12.1  12:46.59 ruby1.8            
14087 www-data  16   0  241m 117m 2328 S   21 11.4   3:41.04 ruby1.8            
15883 www-data  16   0  239m 115m 2328 S    0 11.3   1:35.61 ruby1.8            
share|improve this question
    
Please post the entire /var/log/messages or the output of grep -C 5 oom-killer /var/log/messages. –  Aleksandr Levchuk Dec 29 '10 at 5:13

3 Answers 3

Check the log messages for indications of the kernel out-of-memory killer, or OOM killed in the output of dmesg. That may give some indication of which process(es) were the target of the OOM killer. Also take a look at the following:

http://lwn.net/Articles/317814/

and

http://linux-mm.org/OOM_Killer

What does this system do? Are you exhausting swap at the same time? It looks like rsyslogd is the issue, based on your external link detailing the crash. This could be a situation where a periodic restart of the app would be handy.

share|improve this answer
    
Seems to be a known rsyslogd bug never solved (read the last post). bugs.debian.org/cgi-bin/bugreport.cgi?bug=509292 I'd advise to report it again. –  tmow Dec 29 '10 at 8:11
    
As a quick work-around, would it be a good idea to set up a cronjob to restart apache weekly? So that it doesn't need to use OOM killer. –  nbolton Dec 29 '10 at 12:54

2.6.18 is a very old kernel. I've run into problems where certain conditions can trigger infinite loops in the kernel, resulting in anything from memory exhaustion to I/O bandwidth being entirely used up flushing the same data to disk in an endless loop (which causes load spikes but normal CPU use.)

These bugs tend to get fixed soon after being reported, so a kernel upgrade is an easy fix for this - plus upgrading the kernel means you get some security fixes thrown in for free :-)

share|improve this answer
    
He's using the RHEL5 kernel which has a lot of patches on top of vanilla 2.6.18, and gets security updates from the vendor. That being said, 2.6.18-164.9.1 is old and has several more or less serious vulnerabilities (newest EL5 kernel ATM being 2.6.18-194.26.1). –  janneb Dec 29 '10 at 13:29

On another note, don't forget that Cacti and the like graph at a certain resolution ( collectd is 5s by default, cacti I believe 30s by default ) so you have a period of 30-60 seconds that do not necessarily show up on your graphs ... if the system is totally bogged down, this will also affect the data collection daemon.

You may find additional useful information within your log files be they general /var/log/messages or service specific /var/log/apache2/error.log.

If you cannot, then I'd recommend you go over your services ( I noticed apache2 within your log extract above ) and verify whether they are capable of causing a memory exhaustion situation on your server. ( ex.: default apache configuration, with mod_prefork and php should be capable of bringing your system to a halt ).

share|improve this answer
    
If passenger is causing you to run out of memory you may have an problem as I don't believe there's a way to limit it directly from within passenger. One thing you may want to look at is limiting memory usage through Cgroups via the cgroup-bin package. But this will most certainly require a kernel update on your part. –  delerious010 Dec 29 '10 at 17:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.