what do these messages suggest (linux kernel dump, seems OOM)?

Question

I got the following errors from system log (/var/log/messages). what they indicate? And how to solve them?

Sep 30 05:19:35 node23 kernel: Mem-info:
Sep 30 05:19:35 node23 kernel: Node 0 DMA per-cpu:
Sep 30 05:19:35 node23 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 30 05:19:35 node23 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 30 05:19:35 node23 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 30 05:19:35 node23 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 30 05:19:35 node23 kernel: cpu 2 hot: high 0, batch 1 used:0
Sep 30 05:19:35 node23 kernel: cpu 2 cold: high 0, batch 1 used:0
Sep 30 05:19:35 node23 kernel: cpu 3 hot: high 0, batch 1 used:0
Sep 30 05:19:35 node23 kernel: cpu 3 cold: high 0, batch 1 used:0
Sep 30 05:19:35 node23 kernel: Node 0 DMA32 per-cpu:
Sep 30 05:19:35 node23 kernel: cpu 0 hot: high 186, batch 31 used:64
Sep 30 05:19:35 node23 kernel: cpu 0 cold: high 62, batch 15 used:47
Sep 30 05:19:35 node23 kernel: cpu 1 hot: high 186, batch 31 used:52
Sep 30 05:19:35 node23 kernel: cpu 1 cold: high 62, batch 15 used:50
Sep 30 05:19:35 node23 kernel: cpu 2 hot: high 186, batch 31 used:76
Sep 30 05:19:35 node23 kernel: cpu 2 cold: high 62, batch 15 used:23
Sep 30 05:19:35 node23 kernel: cpu 3 hot: high 186, batch 31 used:116
Sep 30 05:19:35 node23 kernel: cpu 3 cold: high 62, batch 15 used:50
Sep 30 05:19:35 node23 kernel: Node 0 Normal per-cpu:
Sep 30 05:19:35 node23 kernel: cpu 0 hot: high 186, batch 31 used:12
Sep 30 05:19:35 node23 kernel: cpu 0 cold: high 62, batch 15 used:15
Sep 30 05:19:35 node23 kernel: cpu 1 hot: high 186, batch 31 used:7
Sep 30 05:19:35 node23 kernel: cpu 1 cold: high 62, batch 15 used:24
Sep 30 05:19:35 node23 kernel: cpu 2 hot: high 186, batch 31 used:93
Sep 30 05:19:35 node23 kernel: cpu 2 cold: high 62, batch 15 used:9
Sep 30 05:19:35 node23 kernel: cpu 3 hot: high 186, batch 31 used:30
Sep 30 05:19:35 node23 kernel: cpu 3 cold: high 62, batch 15 used:47
Sep 30 05:19:35 node23 kernel: Node 0 HighMem per-cpu: empty
Sep 30 05:19:35 node23 kernel: Free pages:       68484kB (0kB HighMem)
Sep 30 05:19:35 node23 kernel: Active:3469506 inactive:0 dirty:0 writeback:0 unstable:0 free:17121 slab:19849 mapped-file:3469691 mapped-anon:3 pagetables:7393
Sep 30 05:19:35 node23 kernel: Node 0 DMA free:11064kB min:8kB low:8kB high:12kB active:0kB inactive:0kB present:10672kB pages_scanned:0 all_unreclaimable? yes
Sep 30 05:19:35 node23 kernel: lowmem_reserve[]: 0 3254 13859 13859
Sep 30 05:19:35 node23 kernel: Node 0 DMA32 free:45932kB min:3536kB low:4420kB high:5304kB active:3209680kB inactive:0kB present:3332896kB pages_scanned:21228488 all_unreclaimable? yes
Sep 30 05:19:35 node23 kernel: lowmem_reserve[]: 0 0 10605 10605
Sep 30 05:19:35 node23 kernel: Node 0 Normal free:11488kB min:11520kB low:14400kB high:17280kB active:10668472kB inactive:0kB present:10859520kB pages_scanned:22327217 all_unreclaimable? yes
Sep 30 05:19:35 node23 kernel: lowmem_reserve[]: 0 0 0 0
Sep 30 05:19:35 node23 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Sep 30 05:19:35 node23 kernel: lowmem_reserve[]: 0 0 0 0
Sep 30 05:19:35 node23 kernel: Node 0 DMA: 4*4kB 5*8kB 2*16kB 5*32kB 5*64kB 2*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 11064kB
Sep 30 05:19:35 node23 kernel: Node 0 DMA32: 1*4kB 13*8kB 0*16kB 2*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 11*4096kB = 45932kB
Sep 30 05:19:35 node23 kernel: Node 0 Normal: 16*4kB 2*8kB 1*16kB 0*32kB 0*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 11488kB
Sep 30 05:19:35 node23 kernel: Node 0 HighMem: empty
Sep 30 05:19:35 node23 kernel: 3469855 pagecache pages
Sep 30 05:19:35 node23 kernel: Swap cache: add 246644, delete 246644, find 60830/64691, race 0+6
Sep 30 05:19:35 node23 kernel: Free swap  = 9933620kB
Sep 30 05:19:35 node23 kernel: Total swap = 9976356kB
Sep 30 05:19:35 node23 kernel: Free swap:       9933620kB
Sep 30 05:19:35 node23 kernel: 3801088 pages of RAM
Sep 30 05:19:35 node23 kernel: 272953 reserved pages
Sep 30 05:19:35 node23 kernel: 3470343 pages shared
Sep 30 05:19:35 node23 kernel: 0 pages swap cached
Sep 30 05:19:35 node23 kernel: Out of memory: Killed process 2715 (dbus-daemon).
Sep 30 05:19:35 node23 kernel: irqbalance invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0

Answer 1

So, the oom killer killed the process, as a last ditch effort because there was not enough ram. However, the real question is where did your ram go. Pay close attention to top and your logs which might give you some actionable information on preventing this going forward.

Answer 2

oom-killer killer dbus-daemon. More information - http://linux-mm.org/OOM.

Answer 3

You will be well served to get a tool that will tell you when the server is getting close to a memory high water mark as well tell you which process or precesses are consuming memory.

Example; if you don't have Apache configured the way you want it will create a lot of httpd processes which will eat up your memory and force the OS to OOM reaction.

Check out an advanced monitoring solution: appfirst.com cloudkick.com

You should verify how much swap space you have. Use the command % df -h to see how much swap you have. The fastest way to add swap if you need it is to use a swap file. Here are a few quick steps:

% dd if=/dev/zero of=/swapfile bs=1024 count=1048576
% sudo mkswap
/etc/fstab entry:  /swapfile       none    swap    sw      0       0
% mount -a