2

Can you please tell me what cause this Oracle process killed? Seems like plenty of RAM free, and plenty of SWAP free. There followed few other oracle processes killed. The VM has the 16G of vMem and 8 vCPU. But I am posting here the first oracle process that got killed:

 Mar  1 20:00:58 ******* kernel: oracle invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Mar  1 20:00:58 ******* kernel: oracle cpuset=/ mems_allowed=0
Mar  1 20:00:58 ******* kernel: Pid: 2370, comm: oracle Not tainted 2.6.32-431.el6.x86_64 #1
Mar  1 20:00:58 ******* kernel: Call Trace:
Mar  1 20:00:58 ******* kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0
Mar  1 20:00:58 ******* kernel: [] ? dump_header+0x90/0x1b0
Mar  1 20:00:58 ******* kernel: [] ? security_real_capable_noaudit+0x3c/0x70
Mar  1 20:00:58 ******* kernel: [] ? oom_kill_process+0x82/0x2a0
Mar  1 20:00:58 ******* kernel: [] ? select_bad_process+0xe1/0x120
Mar  1 20:00:58 ******* kernel: [] ? out_of_memory+0x220/0x3c0
Mar  1 20:00:58 ******* kernel: [] ? __alloc_pages_nodemask+0x8ac/0x8d0
Mar  1 20:00:58 ******* kernel: [] ? alloc_pages_vma+0x9a/0x150
Mar  1 20:00:58 ******* kernel: [] ? handle_pte_fault+0x73d/0xb00
Mar  1 20:00:58 ******* kernel: [] ? free_pgtables+0xce/0x120
Mar  1 20:00:58 ******* kernel: [] ? unmap_region+0xcd/0x130
Mar  1 20:00:58 ******* kernel: [] ? vma_prio_tree_add+0x75/0xd0
Mar  1 20:00:58 ******* kernel: [] ? handle_mm_fault+0x22a/0x300
Mar  1 20:00:58 ******* kernel: [] ? __do_page_fault+0x138/0x480
Mar  1 20:00:58 ******* kernel: [] ? do_mmap_pgoff+0x335/0x380
Mar  1 20:00:58 ******* kernel: [] ? do_page_fault+0x3e/0xa0
Mar  1 20:00:58 ******* kernel: [] ? page_fault+0x25/0x30
Mar  1 20:00:58 ******* kernel: Mem-Info:
Mar  1 20:00:58 ******* kernel: Node 0 DMA per-cpu:
Mar  1 20:00:58 ******* kernel: CPU    0: hi:    0, btch:   1 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    1: hi:    0, btch:   1 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    2: hi:    0, btch:   1 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    3: hi:    0, btch:   1 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    4: hi:    0, btch:   1 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    5: hi:    0, btch:   1 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    6: hi:    0, btch:   1 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    7: hi:    0, btch:   1 usd:   0
Mar  1 20:00:58 ******* kernel: Node 0 DMA32 per-cpu:
Mar  1 20:00:58 ******* kernel: CPU    0: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    1: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    2: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    3: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    4: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    5: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    6: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    7: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: Node 0 Normal per-cpu:
Mar  1 20:00:58 ******* kernel: CPU    0: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    1: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    2: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    3: hi:  186, btch:  31 usd:  20
Mar  1 20:00:58 ******* kernel: CPU    4: hi:  186, btch:  31 usd:  32
Mar  1 20:00:58 ******* kernel: CPU    5: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: CPU    6: hi:  186, btch:  31 usd: 184
Mar  1 20:00:58 ******* kernel: CPU    7: hi:  186, btch:  31 usd:   0
Mar  1 20:00:58 ******* kernel: active_anon:2673615 inactive_anon:368657 isolated_anon:0
Mar  1 20:00:58 ******* kernel: active_file:3541 inactive_file:3962 isolated_file:32
Mar  1 20:00:58 ******* kernel: unevictable:0 dirty:3 writeback:2770 unstable:0
Mar  1 20:00:58 ******* kernel: free:33763 slab_reclaimable:16555 slab_unreclaimable:28221
Mar  1 20:00:58 ******* kernel: mapped:1517627 shmem:1730877 pagetables:906135 bounce:0
Mar  1 20:00:58 ******* kernel: Node 0 DMA free:15132kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14740kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar  1 20:00:58 ******* kernel: lowmem_reserve[]: 0 3000 16130 16130
Mar  1 20:00:58 ******* kernel: Node 0 DMA32 free:64904kB min:12556kB low:15692kB high:18832kB active_anon:2064816kB inactive_anon:516452kB active_file:492kB inactive_file:188kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:2319432kB shmem:2352892kB slab_reclaimable:7420kB slab_unreclaimable:3620kB kernel_stack:832kB pagetables:24672kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1 all_unreclaimable? no
Mar  1 20:00:58 ******* kernel: lowmem_reserve[]: 0 0 13130 13130
Mar  1 20:00:58 ******* kernel: Node 0 Normal free:55016kB min:54964kB low:68704kB high:82444kB active_anon:8629644kB inactive_anon:958176kB active_file:13672kB inactive_file:15660kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:13445120kB mlocked:0kB dirty:12kB writeback:11080kB mapped:3751076kB shmem:4570616kB slab_reclaimable:58800kB slab_unreclaimable:109264kB kernel_stack:5360kB pagetables:3599868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
Mar  1 20:00:58 ******* kernel: lowmem_reserve[]: 0 0 0 0
Mar  1 20:00:58 ******* kernel: Node 0 DMA: 3*4kB 2*8kB 2*16kB 3*32kB 2*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 3*4096kB = 15132kB
Mar  1 20:00:58 ******* kernel: Node 0 DMA32: 1225*4kB 859*8kB 878*16kB 547*32kB 184*64kB 34*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 65596kB
Mar  1 20:00:58 ******* kernel: Node 0 Normal: 9165*4kB 1804*8kB 46*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 55924kB
Mar  1 20:00:58 ******* kernel: 1760824 total pagecache pages
Mar  1 20:00:58 ******* kernel: 22460 pages in swap cache
Mar  1 20:00:58 ******* kernel: Swap cache stats: add 6636857, delete 6614397, find 15635455/16141480
Mar  1 20:00:58 ******* kernel: Free swap  = 33548340kB
Mar  1 20:00:58 ******* kernel: Total swap = 36184056kB
Mar  1 20:00:58 ******* kernel: 4194288 pages RAM
Mar  1 20:00:58 ******* kernel: 111808 pages reserved
Mar  1 20:00:58 ******* kernel: 59252583 pages shared
Mar  1 20:00:58 ******* kernel: 2502605 pages non-shared
Mar  1 20:00:58 ******* kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Mar  1 20:00:58 ******* kernel: [  612]     0   612     2769       42   2     -17         -1000 udevd
Mar  1 20:00:58 ******* kernel: [ 1872]     0  1872    47365      204   7       0             0 vmtoolsd
Mar  1 20:00:58 ******* kernel: [ 1980]     0  1980    23294      109   6     -17         -1000 auditd
Mar  1 20:00:58 ******* kernel: [ 1996]     0  1996    62898      842   4       0             0 rsyslogd
Mar  1 20:00:58 ******* kernel: [ 2025]     0  2025     2738       93   3       0             0 irqbalance
Mar  1 20:00:58 ******* kernel: [ 2039]    32  2039     4744       68   4       0             0 rpcbind
Mar  1 20:00:58 ******* kernel: [ 2071]    29  2071     5837       61   3       0             0 rpc.statd
Mar  1 20:00:58 ******* kernel: [ 2092]     0  2092     5773       31   1       0             0 rpc.idmapd
Mar  1 20:00:58 ******* kernel: [ 2211]     0  2211    39323      127   5       0             0 pbx_exchange
Mar  1 20:00:58 ******* kernel: [ 2223]     0  2223    48106      158   5       0             0 winbindd
Mar  1 20:00:58 ******* kernel: [ 2237]     0  2237     1020       48   4       0             0 acpid
Mar  1 20:00:58 ******* kernel: [ 2323]     0  2323    49766      281   0       0             0 winbindd
Mar  1 20:00:58 ******* kernel: [ 2540]     0  2540    26827       11   5       0             0 rpc.rquotad
Mar  1 20:00:58 ******* kernel: [ 2544]     0  2544     5414       41   5       0             0 rpc.mountd
Mar  1 20:00:58 ******* kernel: [ 2580]     0  2580     1570       23   0       0             0 mcelog
Mar  1 20:00:58 ******* kernel: [ 2592]     0  2592    16651       78   5     -17         -1000 sshd
Mar  1 20:00:58 ******* kernel: [ 2600]     0  2600     5545      105   3       0             0 xinetd
Mar  1 20:00:58 ******* kernel: [ 2608]    38  2608     7147      132   5       0             0 ntpd
Mar  1 20:00:58 ******* kernel: [ 2618]   498  2618    25741       57   2       0             0 uuidd
Mar  1 20:00:58 ******* kernel: [ 2630]     0  2630    43170      139   3       0             0 vnetd
Mar  1 20:00:58 ******* kernel: [ 2638]     0  2638    52398      158   2       0             0 bpcd
Mar  1 20:00:58 ******* kernel: [ 2655]     0  2655   198335      478   4       0             0 nbdisco
Mar  1 20:00:58 ******* kernel: [ 2676]     0  2676    76958       82   2       0             0 mtstrmd
Mar  1 20:00:58 ******* kernel: [ 2707]     0  2707    22314      141   0       0             0 sendmail
Mar  1 20:00:58 ******* kernel: [ 2716]    51  2716    19658       80   4       0             0 sendmail
Mar  1 20:00:58 ******* kernel: [ 2734]     0  2734   200856      353   7       0             0 avagent.bin
Mar  1 20:00:58 ******* kernel: [ 2747]     0  2747    44287      178   3       0             0 tuned
Mar  1 20:00:58 ******* kernel: [ 2757]     0  2757    29333      103   6       0             0 crond
Mar  1 20:00:58 ******* kernel: [ 2778]     0  2778    27431      167   7       0             0 saphostexec
Mar  1 20:00:58 ******* kernel: [ 2805]   600  2805   545016     4031   5       0             0 sapstartsrv
Mar  1 20:00:58 ******* kernel: [ 2885]   834  2885   100602      294   3       0             0 sapstartsrv
Mar  1 20:00:58 ******* kernel: [ 2904]     0  2904     5385       31   6       0             0 atd
Mar  1 20:00:58 ******* kernel: [ 2928]     0  2928    26005       69   5       0             0 rhsmcertd
Mar  1 20:00:58 ******* kernel: [ 2935]     0  2935     8154     1110   0       0             0 saposcol
Mar  1 20:00:58 ******* kernel: [ 3098]   834  3098    13538       50   3       0             0 sapstart
Mar  1 20:00:58 ******* kernel: [ 3128]   834  3128    43278      119   5       0             0 jc.sapDAA_SMDA9
Mar  1 20:00:58 ******* kernel: [ 3144]   834  3144  1276839    57796   4       0             0 jstart
Mar  1 20:00:58 ******* kernel: [ 3211]   703  3211    33752      378   5       0             0 perl
Mar  1 20:00:58 ******* kernel: [ 3288]   703  3288  1181563    62355   0       0             0 java
Mar  1 20:00:58 ******* kernel: [ 3497]     0  3497     1016       34   1       0             0 mingetty
Mar  1 20:00:58 ******* kernel: [ 3499]     0  3499     1016       34   1       0             0 mingetty
Mar  1 20:00:58 ******* kernel: [ 3502]     0  3502     1016       34   1       0             0 mingetty
Mar  1 20:00:58 ******* kernel: [ 3504]     0  3504     1016       34   2       0             0 mingetty
Mar  1 20:00:58 ******* kernel: [ 3506]     0  3506     1016       34   1       0             0 mingetty
Mar  1 20:00:58 ******* kernel: [ 3508]     0  3508     1016       34   1       0             0 mingetty
Mar  1 20:00:58 ******* kernel: [ 3515]     0  3515     3098       41   2     -17         -1000 udevd
Mar  1 20:00:58 ******* kernel: [ 3516]     0  3516     3098       41   4     -17         -1000 udevd
Mar  1 20:00:58 ******* kernel: [13764]     0 13764    48089       89   7       0             0 winbindd
Mar  1 20:00:58 ******* kernel: [13765]     0 13765    48089       92   7       0             0 winbindd
Mar  1 20:00:58 ******* kernel: [13873]   703 13873  2403434     6196   5       0             0 oracle
Mar  1 20:00:58 ******* kernel: [13875]   703 13875  2402873      651   3       0             0 oracle
Mar  1 20:00:58 ******* kernel: [13880]   703 13880  2402873      423   4       0             0 oracle
Mar  1 20:00:58 ******* kernel: [13875]   703 13875  2402873      651   3       0             0 oracle
Mar  1 20:00:58 ******* kernel: [13880]   703 13880  2402873      423   4       0             0 oracle

.. Note: Removed bunch of oracle processes here so as to limit the character length for the posting here. Total of 296 oracle process running.
..
Mar  1 20:00:59 ******* kernel: [18644]     0 18644    44207      371   1       0             0 bpclntcmd
Mar  1 20:00:59 ******* kernel: [18647]   703 18647    57442      240   3       0             0 oracle
Mar  1 20:00:59 ******* kernel: [18656]   703 18656    57442      185   6       0             0 oracle
Mar  1 20:00:59 ******* kernel: [18657] 54329 18657     9279      196   1       0             0 nrpe
Mar  1 20:00:59 ******* kernel: [18660] 54329 18660     9314      255   2       0             0 nrpe
Mar  1 20:00:59 ******* kernel: [18662]     0 18662    39263      289   5       0             0 crond
Mar  1 20:00:59 ******* kernel: [18663]     0 18663     5745      341   1       0             0 saposcol
Mar  1 20:00:59 ******* kernel: [18664] 54329 18664     9315      146   3       0             0 nrpe
Mar  1 20:00:59 ******* kernel: [18665] 54329 18665     5730       76   0       0             0 check_open_file
Mar  1 20:00:59 ******* kernel: [18666] 54329 18666     6611      191   4       0             0 xinetd
Mar  1 20:00:59 ******* kernel: [18667]     0 18667     8389      183   1       0             0 sapcimb
Mar  1 20:00:59 ******* kernel: [18669]     0 18669     6610      171   0       0             0 xinetd
Mar  1 20:00:59 ******* kernel: [18670]     0 18670     6610      171   0       0             0 xinetd
Mar  1 20:00:59 ******* kernel: [18677]     0 18677     6610      177   5       0             0 xinetd
Mar  1 20:00:59 ******* kernel: [18678]   703 18678    29497      275   4       0             0 perl
Mar  1 20:00:59 ******* kernel: [18682]   703 18682    29497      252   7       0             0 perl
Mar  1 20:00:59 ******* kernel: [18683]   703 18683    29497      231   0       0             0 perl
Mar  1 20:00:59 ******* kernel: [18687]     0 18687     2620       92   1       0             0 .SAPOSCOL_00000
Mar  1 20:00:59 ******* kernel: [18688]     0 18688     6610      186   5       0             0 xinetd
Mar  1 20:00:59 ******* kernel: [18689]     0 18689     6610      189   2       0             0 xinetd
Mar  1 20:00:59 ******* kernel: [18690]     0 18690     6610      191   3       0             0 xinetd
Mar  1 20:00:59 ******* kernel: [18691]     0 18691     6610      194   2       0             0 xinetd
Mar  1 20:00:59 ******* kernel: Out of memory: Kill process 13900 (oracle) score 77 or sacrifice child
Mar  1 20:00:59 ******* kernel: Killed process 13900, UID 703, (oracle) total-vm:9622308kB, anon-rss:5180kB, file-rss:4028040kB
 

From above, I think these lines says I have plenty of RAM and swap. right?:

Node 0 DMA free:15132kB
Node 0 DMA32 free:64904kB
Node 0 Normal free:55016kB
Free swap  = 33548340kB
Total swap = 36184056kB

Wondering what does it mean by "all_unreclaimable? yes" for Node 0 DMA, and "all_unreclaimable? no"Node 0 DMA31 and Node 0 Normal !

Also, here are the info that might give more info about the server settings:

 $sudo sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.msgmni = 1024
kernel.sem = 1250 256000 100 8192
vm.max_map_count = 1000000
kernel.shmall = 1152921504606846720
fs.file-max = 19801952
net.core.rmem_default = 1048576
net.core.wmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
vm.swappiness = 0
vm.dirty_background_ratio = 3
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100
kernel.shmmni = 4096
2

You don't have much free memory at all.

First, vm.swappiness = 0 Only do this if you are definitely sure you have enough. Setting it low to 10 or so might prevent an out of memory condition. And will actually make use of your paging space.

From the summary of node 0, your 16 GB is roughly a quarter page tables, a quarter shared memory, half anonymous program memory, and some various odds and ends. Notice that the readily available file memory, plus free, is only tens of MB, not large. It won't be able to give you another GB or so of shared memory.

Page tables are eating you alive. You may not have huge pages enabled, which Oracle recommends for databases, and Red Hat does too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.