0

I am running into oom_killer activity and cannot undesrand dmesg output

[Fri Jan 11 02:11:48 2019] lowmem_reserve[]: 0 2811 64249 64249
[Fri Jan 11 02:11:48 2019] Node 0 DMA32 free:248704kB min:2956kB low:3692kB high:4432kB active_anon:2031516kB inactive_anon:524376kB active_file:160kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2881076kB mlocked:0kB dirty:0kB writeback:0kB mapped:2564kB shmem:56560kB slab_reclaimable:44392kB slab_unreclaimable:9560kB kernel_stack:3904kB pagetables:9288kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:427 all_unreclaimable? yes
[Fri Jan 11 02:11:48 2019] lowmem_reserve[]: 0 0 61437 61437
[Fri Jan 11 02:11:48 2019] Node 0 Normal free:64472kB min:64608kB low:80760kB high:96912kB active_anon:59250028kB inactive_anon:2533808kB active_file:5060kB inactive_file:8272kB unevictable:0kB isolated(anon):0kB isolated(file):352kB present:63963136kB managed:62912504kB mlocked:0kB dirty:0kB writeback:4kB mapped:72292kB shmem:1480340kB slab_reclaimable:245132kB slab_unreclaimable:99196kB kernel_stack:29440kB pagetables:217044kB unstable:0kB bounce:0kB free_pcp:188kB local_pcp:188kB free_cma:0kB writeback_tmp:0kB pages_scanned:37993 all_unreclaimable? yes
[Fri Jan 11 02:11:48 2019] lowmem_reserve[]: 0 0 0 0
[Fri Jan 11 02:11:48 2019] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15860kB
[Fri Jan 11 02:11:48 2019] Node 0 DMA32: 723*4kB (UEM) 544*8kB (UEM) 728*16kB (UEM) 820*32kB (UEM) 714*64kB (UEM) 441*128kB (UEM) 233*256kB (UEM) 64*512kB (UEM) 9*1024kB (UEM) 0*2048kB 0*4096kB = 248908kB
[Fri Jan 11 02:11:48 2019] Node 0 Normal: 12711*4kB (UEM) 1791*8kB (UEM) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65188kB
[Fri Jan 11 02:11:48 2019] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Fri Jan 11 02:11:48 2019] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Fri Jan 11 02:11:48 2019] 420932 total pagecache pages
[Fri Jan 11 02:11:48 2019] 33002 pages in swap cache
[Fri Jan 11 02:11:48 2019] Swap cache stats: add 1061688, delete 1028686, find 21081912/21097467
[Fri Jan 11 02:11:48 2019] Free swap  = 0kB
[Fri Jan 11 02:11:48 2019] Total swap = 2097148kB
[Fri Jan 11 02:11:48 2019] 16777086 pages RAM
[Fri Jan 11 02:11:48 2019] 0 pages HighMem/MovableOnly
[Fri Jan 11 02:11:48 2019] 324714 pages reserved
[Fri Jan 11 02:11:48 2019] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[Fri Jan 11 02:11:48 2019] [  633]     0   633    46199    30241      97       47             0 systemd-journal
[Fri Jan 11 02:11:48 2019] [  675]     0   675    12641      143      26       40         -1000 systemd-udevd
[Fri Jan 11 02:11:48 2019] [  821]     0   821   119828       16      36      147             0 lvmetad
[Fri Jan 11 02:11:48 2019] [ 1315]     0  1315    14423       35      28       83         -1000 auditd
[Fri Jan 11 02:11:48 2019] [ 1318]     0  1318     5927        0      16       55             0 rpc.idmapd
[Fri Jan 11 02:11:48 2019] [ 1339]   169  1339     3768        8      13       36             0 lsmd
[Fri Jan 11 02:11:48 2019] [ 1340]    81  1340     7766       64      21       56          -900 dbus-daemon
[Fri Jan 11 02:11:48 2019] [ 1341]     0  1341    25997       84      45      323             0 VGAuthService
[Fri Jan 11 02:11:48 2019] [ 1342]     0  1342    58916      119      60      238             0 vmtoolsd
[Fri Jan 11 02:11:48 2019] [ 1343]     0  1343     1618       23       9       19             0 rngd
[Fri Jan 11 02:11:48 2019] [ 1344]     0  1344     5520       28      17       40             0 systemd-logind
[Fri Jan 11 02:11:48 2019] [ 1345]   997  1345   135183      304      62     1078             0 polkitd
[Fri Jan 11 02:11:48 2019] [ 1347]     0  1347    63153      417      30       86             0 rsyslogd
[Fri Jan 11 02:11:48 2019] [ 1350]     0  1350     5409       43      15       41             0 irqbalance
[Fri Jan 11 02:11:48 2019] [ 1391]     0  1391    67265       16      43      639             0 gssproxy
[Fri Jan 11 02:11:48 2019] [ 1408]    38  1408     8570       51      22      110             0 ntpd
[Fri Jan 11 02:11:48 2019] [ 1471]     0  1471    10296        2      23      100             0 rpc.gssd
[Fri Jan 11 02:11:48 2019] [ 1483]     0  1483     8099        5      21       56             0 atd
[Fri Jan 11 02:11:48 2019] [ 1486]     0  1486    33193       35      22      132             0 crond
[Fri Jan 11 02:11:48 2019] [ 1489]     0  1489    29146        2      14       38             0 agetty
[Fri Jan 11 02:11:48 2019] [ 1622]     0  1622   141697      171      96     2507             0 tuned
[Fri Jan 11 02:11:48 2019] [ 1624]     0  1624    72433      412      96     3258             0 salt-minion
[Fri Jan 11 02:11:48 2019] [ 1629]     0  1629    49494        0      52      288             0 cupsd
[Fri Jan 11 02:11:48 2019] [ 1632]    29  1632    10594        2      25      206             0 rpc.statd
[Fri Jan 11 02:11:48 2019] [ 1638]     0  1638    28910       22      14       19             0 rhsmcertd
[Fri Jan 11 02:11:48 2019] [ 1639]     0  1639    27059       27      55      220         -1000 sshd
[Fri Jan 11 02:11:48 2019] [ 1649]     0  1649     6248        0      16       58             0 xinetd
[Fri Jan 11 02:11:48 2019] [ 1671]    32  1671    16265       55      36       96             0 rpcbind
[Fri Jan 11 02:11:48 2019] [ 1701]     0  1701    10655      110      23      107             0 rpc.mountd
[Fri Jan 11 02:11:48 2019] [ 1710]     0  1710    42838      110      30      141             0 pbx_exchange
[Fri Jan 11 02:11:48 2019] [ 2118]     0  2118    46898       84      37       95             0 vnetd
[Fri Jan 11 02:11:48 2019] [ 2398]     0  2398    37270     1007      49       92             0 adclient
[Fri Jan 11 02:11:48 2019] [ 2399]     0  2399    24721       38      19       81             0 cdcwatch
[Fri Jan 11 02:11:48 2019] [ 2547]     0  2547    54892      113      50      191             0 bpcd
[Fri Jan 11 02:11:48 2019] [ 2688]     0  2688    22946       81      44      182             0 master
[Fri Jan 11 02:11:48 2019] [ 2714]    89  2714    23017       38      48      230             0 qmgr
[Fri Jan 11 02:11:48 2019] [ 3250]     0  3250   189579      431     122      906             0 nbdisco
[Fri Jan 11 02:11:48 2019] [ 3723]     0  3723   195675     8308     156     1030             0 salt-minion
[Fri Jan 11 02:11:48 2019] [ 4057]     0  4057    22743       13      14       38             0 nqmagt
[Fri Jan 11 02:11:48 2019] [ 4062]     0  4062   135901     1057      46      750             0 agent
[Fri Jan 11 02:11:48 2019] [ 4069]     0  4069   469383     1598     119     3347             0 VigilEntAgent
[Fri Jan 11 02:11:48 2019] [ 4337]     0  4337   102252      564     112     4317             0 salt-minion
[Fri Jan 11 02:11:48 2019] [23762]     0 23762     4095      229       7       74             0 symcfgd
[Fri Jan 11 02:11:48 2019] [23795]     0 23795    90053    82831     186      682             0 rtvscand
[Fri Jan 11 02:11:48 2019] [23866]     0 23866    37105     2054      43     3179             0 smcd
[Fri Jan 11 02:11:48 2019] [29541] 57317 29541    37128        4      31       75             0 ps
[Fri Jan 11 02:11:48 2019] [ 2842] 57317  2842    37128        5      29       74             0 ps
[Fri Jan 11 02:11:48 2019] [30378] 57317 30378    37128       11      31       67             0 ps
[Fri Jan 11 02:11:48 2019] [27560] 57317 27560    28312        3      13       68             0 bash
[Fri Jan 11 02:11:48 2019] [30652] 57317 30652    28284        5      14       59             0 bash
[Fri Jan 11 02:11:48 2019] [30665] 57317 30665  2599863   271425     886    63297             0 java
[Fri Jan 11 02:11:48 2019] [ 4753] 57317  4753    28313       10      13       61             0 bash
[Fri Jan 11 02:11:48 2019] [ 7977] 57317  7977    28285        8      14       57             0 bash
[Fri Jan 11 02:11:48 2019] [ 7990] 57317  7990  2599863   327013     930    32865             0 java
[Fri Jan 11 02:11:48 2019] [ 8440] 57317  8440    28313       11      12       62             0 bash
[Fri Jan 11 02:11:48 2019] [ 8829] 57317  8829    28285       12      12       54             0 bash
[Fri Jan 11 02:11:48 2019] [ 8842] 57317  8842  2599863   299812     952    72150             0 java
[Fri Jan 11 02:11:48 2019] [ 8451] 57317  8451    37128       79      31        0             0 ps
[Fri Jan 11 02:11:48 2019] [ 2876] 57317  2876    37128       79      29        0             0 ps
[Fri Jan 11 02:11:48 2019] [19806] 57317 19806    37129       79      29        0             0 ps
[Fri Jan 11 02:11:48 2019] [24141] 57317 24141    37129       78      30        0             0 ps
[Fri Jan 11 02:11:48 2019] [22801] 57317 22801    37128       79      30        0             0 ps
[Fri Jan 11 02:11:48 2019] [20694] 57317 20694    37128       79      30        0             0 ps
[Fri Jan 11 02:11:48 2019] [13828] 57317 13828    37128       78      29        0             0 ps
[Fri Jan 11 02:11:48 2019] [ 3884] 57317  3884    37129       79      29        0             0 ps
[Fri Jan 11 02:11:48 2019] [16585] 57317 16585    37128       79      31        0             0 ps
[Fri Jan 11 02:11:48 2019] [10773] 57317 10773    37128       79      31        0             0 ps
[Fri Jan 11 02:11:48 2019] [  984] 57317   984    37129       78      29        0             0 ps
[Fri Jan 11 02:11:48 2019] [26228] 57317 26228    37128       79      30        0             0 ps
[Fri Jan 11 02:11:48 2019] [28863] 57316 28863 16283613  4609499   17059        0             0 java
[Fri Jan 11 02:11:48 2019] [14614] 57317 14614    37128       78      28        0             0 ps
[Fri Jan 11 02:11:48 2019] [22044] 57317 22044    37128       79      31        0             0 ps
[Fri Jan 11 02:11:48 2019] [28610] 57317 28610    37129       79      30        0             0 ps
[Fri Jan 11 02:11:48 2019] [ 2186] 57317  2186    37129       79      29        0             0 ps
[Fri Jan 11 02:11:48 2019] [14078] 57317 14078     4485       77      10        0             0 wrapper
[Fri Jan 11 02:11:48 2019] [14080] 57317 14080  1598728   119038     475        0             0 java
[Fri Jan 11 02:11:48 2019] [14158] 57317 14158     4492       79      10        0             0 wrapper
[Fri Jan 11 02:11:48 2019] [14160] 57317 14160  1594484   143688     488        0             0 java
[Fri Jan 11 02:11:48 2019] [14238] 57317 14238     4492       77      10        0             0 wrapper
[Fri Jan 11 02:11:48 2019] [14240] 57317 14240  1599042   163554     517        0             0 java
[Fri Jan 11 02:11:48 2019] [14322] 57317 14322     4492       80      10        0             0 wrapper
[Fri Jan 11 02:11:48 2019] [14324] 57317 14324  1598785   158818     529        0             0 java
[Fri Jan 11 02:11:48 2019] [14402] 57317 14402     4492       77      10        0             0 wrapper
[Fri Jan 11 02:11:48 2019] [14405] 57317 14405  1598785   156120     511        0             0 java
[Fri Jan 11 02:11:48 2019] [23366]     0 23366    50567      756      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [23367]     0 23367    14342      101      33        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23368]     0 23368    14342      103      31        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23369]     0 23369    14342       96      31        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23370]     0 23370    14342      101      30        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23371]     0 23371    14342      102      32        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23372]     0 23372    14342      102      33        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23373]     0 23373    14342      102      32        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23374]     0 23374    14342      104      30        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23375]     0 23375    14342      103      31        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23376]     0 23376    14342      103      32        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23377]     0 23377    14342      102      31        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [23378]     0 23378    14342      102      33        0             0 rotatelogs
[Fri Jan 11 02:11:48 2019] [ 5477] 57316  5477  7630798  2313242    9166        0             0 java
[Fri Jan 11 02:11:48 2019] [ 2224]  9618  2224    52716      795      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8329]  9618  8329    52716      795      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [10808]  9618 10808    69100      797      99        0             0 httpd
[Fri Jan 11 02:11:48 2019] [15676] 57316 15676  8922598  1626467    4977        0             0 java
[Fri Jan 11 02:11:48 2019] [30309]    89 30309    22972      254      45        0             0 pickup
[Fri Jan 11 02:11:48 2019] [30581] 57317 30581  5204496   687312    1524        0             0 java
[Fri Jan 11 02:11:48 2019] [30690] 57317 30690  5027787   649120    1415        0             0 java
[Fri Jan 11 02:11:48 2019] [31537] 57317 31537  5204498   775719    1715        0             0 java
[Fri Jan 11 02:11:48 2019] [32658] 57317 32658  5171025   650715    1415        0             0 java
[Fri Jan 11 02:11:48 2019] [ 2292] 57316  2292  7069828  1488690    3871        0             0 java
[Fri Jan 11 02:11:48 2019] [ 2322] 57317  2322  5203973   644654    1405        0             0 java
[Fri Jan 11 02:11:48 2019] [ 3794] 57317  3794    28284       63      12        0             0 sh
[Fri Jan 11 02:11:48 2019] [ 3799] 57317  3799  1072293   165518     445        0             0 java
[Fri Jan 11 02:11:48 2019] [ 4291]  9618  4291    52716      790      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 4310]  9618  4310    52716      787      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 4620]  9618  4620    52716      787      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 4721]  9618  4721    52716      788      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 4847]  9618  4847    52716      789      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 4867]  9618  4867    52716      788      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 5021]  9618  5021    52716      786      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 5072]  9618  5072    52716      789      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 5075]  9618  5075    69100      788      99        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 6618]  9618  6618    52716      788      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 6623]  9618  6623    52716      788      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 6758]  9618  6758    69092      788     100        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 6760]  9618  6760    52716      787      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 6917]  9618  6917    52716      787      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7003]  9618  7003    52716      788      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7787]  9618  7787    52716      787      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7822]  9618  7822    52716      787      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7834]  9618  7834    52716      786      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7837]  9618  7837    52716      787      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7940]  9618  7940    52716      786      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7972]  9618  7972    52716      786      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7974]  9618  7974    52716      786      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 7975]  9618  7975    52716      786      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8215]  9618  8215    52716      778      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8219]  9618  8219    52716      778      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8314]  9618  8314    52716      778      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8360]  9618  8360    69092      778     100        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8361]  9618  8361    69092      780     102        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8362]  9618  8362    52716      778      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8363]  9618  8363    69092      779     100        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8428]  9618  8428    69100      779      99        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8429]  9618  8429    52716      780      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8478]  9618  8478    52716      778      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8482]  9618  8482    52716      778      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8575]  9618  8575    52716      778      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8626]  9618  8626    52716      778      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 8627]  9618  8627    52716      780      98        0             0 httpd
[Fri Jan 11 02:11:48 2019] [ 9707] 57317  9707    28284       60      13        0             0 sh
[Fri Jan 11 02:11:48 2019] [ 9710] 57317  9710  5013780   105123     317        0             0 java
[Fri Jan 11 02:11:48 2019] [ 9771] 57317  9771    28284       61      14        0             0 sh
[Fri Jan 11 02:11:48 2019] [ 9775] 57317  9775  5012346    98369     304        0             0 java
[Fri Jan 11 02:11:48 2019] [ 9820] 57317  9820    28284       60      12        0             0 sh
[Fri Jan 11 02:11:48 2019] [ 9822] 57317  9822  5010296    86077     270        0             0 java
[Fri Jan 11 02:11:48 2019] [ 9859] 57317  9859    37228       89      29        0             0 ps
[Fri Jan 11 02:11:48 2019] [ 9860] 57317  9860    11215       75      25        0             0 ps
[Fri Jan 11 02:11:48 2019] [ 9861] 57317  9861    28276       39      13        0             0 sh
[Fri Jan 11 02:11:48 2019] Out of memory: Kill process 28863 (java) score 272 or sacrifice child
[Fri Jan 11 02:11:48 2019] Killed process 28863 (java) total-vm:65134452kB, anon-rss:18437996kB, file-rss:0kB, shmem-rss:0kB

We are interested in PID 28863

Then why indicated values are different in table and in summary line?

[Fri Jan 11 02:11:48 2019] [28863] 57316 28863 16283613  4609499   17059        0             0 java
[Fri Jan 11 02:11:48 2019] Out of memory: Kill process 28863 (java) score 272 or sacrifice child
[Fri Jan 11 02:11:48 2019] Killed process 28863 (java) total-vm:65134452kB, anon-rss:18437996kB, file-rss:0kB, shmem-rss:0kB

table shows 16283613 of virtual memory while summary says its 65134452kB. Why?

  • total-vm is virtual memory, the process 28863 is using rss memory 4609499, the process that uses more memory, so it's the good to kill and recover the system – c4f4t0r Jan 11 at 14:28
  • But what is the last total-vm? is that the "sum of table"? If so, why OOM killer is triggered? If there is some sort of treashold that if VM>=RAM - go and kill something? or maybe 64GB is magical barier (here it eq to RAM side but maybe it is coincidence) – Antoniossss Jan 11 at 14:41
  • yes, vm-total is total vm allocated by procs, oom kill get triggered when you don't have more ram and swap(Free swap = 0kB), look this anon-rss:18437996kB = 18G of physical memory, if you search for oom-kill you can understand how it's works – c4f4t0r Jan 11 at 14:47
  • But its confusing for me as If I have 18G mem occupied, a.nd there are 64gb physicalyll installed, why OOM it is triggered? I know why this particular process got killed - most oom score - but why it is triggered since there is still RAM available? – Antoniossss Jan 11 at 14:58
  • if you check well, this line is only for a process "Killed process 28863 (java) total-vm:65134452kB, anon-rss:18437996kB, file-rss:0kB, shmem-rss:0kB", that mean only one process is using 18G and try sum the rss of the other java and you will see that you memory is not enough – c4f4t0r Jan 11 at 15:22

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.