6

I am running an ARM router with i2p and tor on it - a Netgear R7000. Of course I've added a full 512 MB of SWAP to it to prevent OOMs, understanding that it could slow down the system... But then I still get OOM killer starting with plenty of SWAP free, and killing tor! More interesting, after killing tor it seems that the system is OK for an unlimited amount of time... But still seems like tor couldn't be swapped. I even tried to turn off the overcommit, didn't helped at all.

See the dmesg log below

resetbutton invoked oom-killer: gfp_mask=0x2000d0, order=0, oom_score_adj=0
CPU: 0 PID: 1500 Comm: resetbutton Tainted: P             3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
 r6:c7a4cdc0 r5:00000000 r4:c6818000 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007c5cc>] (dump_header.isra.13+0x84/0x194)
[<c007c548>] (dump_header.isra.13+0x0/0x194) from [<c007c958>] (oom_kill_process+0x90/0x3e8)
[<c007c8c8>] (oom_kill_process+0x0/0x3e8) from [<c007d17c>] (out_of_memory+0x2c0/0x304)
[<c007cebc>] (out_of_memory+0x0/0x304) from [<c007fcbc>] (__alloc_pages_nodemask+0x5b0/0x694)
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c00a2464>] (cache_alloc_refill+0x2c8/0x600)
[<c00a219c>] (cache_alloc_refill+0x0/0x600) from [<c00a214c>] (kmem_cache_alloc+0x88/0xd8)
[<c00a20c4>] (kmem_cache_alloc+0x0/0xd8) from [<c00afdd4>] (getname_flags+0x2c/0x104)
 r7:b6f5509c r6:00000000 r5:00000000 r4:00020000
[<c00afda8>] (getname_flags+0x0/0x104) from [<c00afec4>] (getname+0x18/0x1c)
[<c00afeac>] (getname+0x0/0x1c) from [<c00a5270>] (do_sys_open+0xbc/0x178)
[<c00a51b4>] (do_sys_open+0x0/0x178) from [<c00a5350>] (SyS_open+0x24/0x28)
 r9:c6818000 r8:c0008d24 r7:00000005 r6:00000000 r5:00000000
r4:00000000
[<c00a532c>] (SyS_open+0x0/0x28) from [<c0008ba0>] (ret_fast_syscall+0x0/0x30)
Mem-info:
Normal per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
HighMem per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
active_anon:10263 inactive_anon:12797 isolated_anon:0
 active_file:957 inactive_file:1335 isolated_file:0
 unevictable:151 dirty:0 writeback:0 unstable:0
 free:13787 slab_reclaimable:912 slab_unreclaimable:5734
 mapped:1171 shmem:0 pagetables:283 bounce:0
 free_cma:0
Normal free:26184kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:96kB inactive_file:64kB unevictable:64kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3648kB slab_unreclaimable:22936kB kernel_stack:1192kB pagetables:1132kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4440 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:28656kB min:128kB low:8772kB high:17416kB active_anon:41052kB inactive_anon:51512kB active_file:3764kB inactive_file:5244kB unevictable:540kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:0kB writeback:0kB mapped:4684kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 102*4kB (UEM) 118*8kB (UEM) 360*16kB (UM) 158*32kB (UM) 28*64kB (UEM) 8*128kB (UM) 4*256kB (UM) 4*512kB (UM) 0*1024kB 0*2048kB 2*4096kB (R) = 26248kB
HighMem: 1177*4kB (UMR) 1620*8kB (UMR) 225*16kB (UMR) 30*32kB (UMR) 29*64kB (UMR) 8*128kB (UMR) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 0*4096kB = 28436kB
9347 total pagecache pages
6914 pages in swap cache
Swap cache stats: add 484419, delete 477505, find 187469/228822
Free swap  = 408788kB
Total swap = 524284kB
65536 pages of RAM
13904 free pages
1661 reserved pages
6646 slab pages
549590 pages shared
6923 pages swap cached
[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  622]     0   622      189        7       3       13             0 hotplug2
[  626]     0   626      199       30       3        6             0 mstpd
[  971]     0   971      217        9       3        8             0 dropbear
[  987]     0   987      297      105       3       30             0 dnsmasq
[ 1112]     0  1112      315       29       4       30             0 ttraff
[ 1195]     0  1195      209       25       3        6             0 dhcp6c
[ 1288]     0  1288      755        3       4       83             0 httpd
[ 1330]     0  1330      315       26       3       18             0 process_monitor
[ 1334]     0  1334      361       44       3       38             0 upnp
[ 1337]     0  1337      334       50       4        9             0 wland
[ 1338]     0  1338      190       42       3        3             0 igmprt
[ 1339]     0  1339      291       10       3        6             0 udhcpc
[ 1449]     0  1449      182       27       3        3             0 cron
[ 1457]     0  1457      370       38       3       32             0 nas
[ 1460]     0  1460      370       36       4       32             0 nas
[ 1495]     0  1495      291       11       3        6             0 syslogd
[ 1497]     0  1497      291       15       3        6             0 klogd
[ 1500]     0  1500      334       32       3       12             0 resetbutton
[ 1684] 65534  1684     1008       18       5      499             0 tinyproxy
[ 1690] 65534  1690     1021        9       5      508             0 tinyproxy
[ 1691] 65534  1691     1021        9       5      508             0 tinyproxy
[ 1692] 65534  1692     1021        9       5      508             0 tinyproxy
[ 1693] 65534  1693     1008        6       5      502             0 tinyproxy
[ 1694] 65534  1694     1008        6       5      502             0 tinyproxy
[ 1695] 65534  1695     1008        6       5      502             0 tinyproxy
[ 1696] 65534  1696     1008        6       5      502             0 tinyproxy
[ 1697] 65534  1697     1008        6       5      502             0 tinyproxy
[ 1698] 65534  1698     1008        6       5      502             0 tinyproxy
[ 1699] 65534  1699     1008        6       5      502             0 tinyproxy
[ 1711]   106  1711    36556    12756      73    10236             0 tor
[ 1730]    13  1730      580        9       4       38             0 polipo
[ 1806]  1000  1806     2828       41       5      124             0 wrapper
[ 1808]  1000  1808    59927     7979      69    13797             0 java
[ 1921]     0  1921      241       18       3       31             0 dropbear
[ 1923]     0  1923      293        9       3        9             0 sh
[ 3705]     0  3705      804        5       4       86             0 bash
[ 5859]     0  5859      320       61       3       12             0 radvd
[ 5860]     0  5860      320       18       3       18             0 radvd
Out of memory: Kill process 1711 (tor) score 118 or sacrifice child
Killed process 1711 (tor) total-vm:146224kB, anon-rss:49036kB, file-rss:1988kB
tor: page allocation failure: order:0, mode:0x20058
CPU: 1 PID: 1711 Comm: tor Tainted: P             3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
 r6:00000000 r5:00000000 r4:00020058 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007def4>] (warn_alloc_failed+0x100/0x110)
[<c007ddf4>] (warn_alloc_failed+0x0/0x110) from [<c007faf0>] (__alloc_pages_nodemask+0x3e4/0x694)
 r3:00040001 r2:00000000
 r7:c0405180 r6:00000001 r5:00000000 r4:00020058
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c007a274>] (find_or_create_page+0x48/0x94)
[<c007a22c>] (find_or_create_page+0x0/0x94) from [<c00cfc08>] (__getblk+0x16c/0x308)
 r9:00000000 r8:0223764c r7:c7418110 r6:00000001 r5:00001000
r4:00000000
[<c00cfa9c>] (__getblk+0x0/0x308) from [<c00d0668>] (__bread+0x1c/0xd8)
[<c00d064c>] (__bread+0x0/0xd8) from [<bf5e96c8>] (_8+0x208/0x4a0 [ext2])
 r4:c6bb5b58
[<bf5e963c>] (_8+0x17c/0x4a0 [ext2]) from [<bf5ea2e0>] (_35+0x78/0x884 [ext2])
[<bf5ea268>] (_35+0x0/0x884 [ext2]) from [<c00d6da8>] (do_mpage_readpage+0x268/0x6d4)
[<c00d6b40>] (do_mpage_readpage+0x0/0x6d4) from [<c00d72d4>] (mpage_readpages+0xc0/0x104)
[<c00d7214>] (mpage_readpages+0x0/0x104) from [<bf5e99ac>] (_61+0x24/0x28 [ext2])
[<bf5e9988>] (_61+0x0/0x28 [ext2]) from [<c0082f08>] (__do_page_cache_readahead+0x170/0x1f8)
[<c0082d98>] (__do_page_cache_readahead+0x0/0x1f8) from [<c0083428>] (ra_submit+0x30/0x38)
[<c00833f8>] (ra_submit+0x0/0x38) from [<c007bb4c>] (filemap_fault+0x1bc/0x434)
[<c007b990>] (filemap_fault+0x0/0x434) from [<c008e5a4>] (__do_fault+0xac/0x490)
[<c008e4f8>] (__do_fault+0x0/0x490) from [<c0091284>] (handle_pte_fault+0x78/0x6d4)
[<c009120c>] (handle_pte_fault+0x0/0x6d4) from [<c0091968>] (__handle_mm_fault+0x88/0x90)
[<c00918e0>] (__handle_mm_fault+0x0/0x90) from [<c00920e0>] (handle_mm_fault+0x28/0x2c)
[<c00920b8>] (handle_mm_fault+0x0/0x2c) from [<c001948c>] (do_page_fault+0x118/0x288)
[<c0019374>] (do_page_fault+0x0/0x288) from [<c0008580>] (do_DataAbort+0x40/0x9c)
[<c0008540>] (do_DataAbort+0x0/0x9c) from [<c0008998>] (__dabt_usr+0x38/0x40)
Exception stack(0xc6bb5fb0 to 0xc6bb5ff8)
5fa0:                                     b6fe43d8 b6cfd864 00000005 00000053
5fc0: 00000001 b6fe43dd b6d0a358 b6cfd864 b866141c 00000000 b6fe43d8 00000000
5fe0: b6d0b658 be8798f0 b6cf2330 b6a4c388 20000010 ffffffff
 r8:b866141c r7:b6cfd864 r6:ffffffff r5:20000010 r4:b6a4c388
Mem-info:
Normal per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
HighMem per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
active_anon:10263 inactive_anon:13171 isolated_anon:0
 active_file:971 inactive_file:1322 isolated_file:0
 unevictable:151 dirty:1 writeback:0 unstable:0
 free:13621 slab_reclaimable:912 slab_unreclaimable:5739
 mapped:1172 shmem:0 pagetables:283 bounce:0
 free_cma:0
Normal free:27136kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:40kB inactive_file:120kB unevictable:64kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3648kB slab_unreclaimable:22956kB kernel_stack:1192kB pagetables:1132kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:27380kB min:128kB low:8772kB high:17416kB active_anon:41052kB inactive_anon:52684kB active_file:3764kB inactive_file:5248kB unevictable:540kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:4kB writeback:0kB mapped:4688kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 96*4kB (UEM) 120*8kB (UEM) 360*16kB (UM) 182*32kB (UM) 32*64kB (UEM) 8*128kB (UM) 4*256kB (UM) 4*512kB (UM) 0*1024kB 0*2048kB 2*4096kB (R) = 27264kB
HighMem: 918*4kB (UMR) 1620*8kB (UMR) 225*16kB (UMR) 30*32kB (UMR) 29*64kB (UMR) 8*128kB (UMR) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 0*4096kB = 27400kB
9533 total pagecache pages
7099 pages in swap cache
Swap cache stats: add 484625, delete 477526, find 187576/228974
Free swap  = 408868kB
Total swap = 524284kB
65536 pages of RAM
13939 free pages
1661 reserved pages
6652 slab pages
549523 pages shared
7099 pages swap cached
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets

The sysctl vm related settings:

vm.admin_reserve_kbytes = 7914
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 0
vm.highmem_is_dirtyable = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 32    32
vm.max_map_count = 65530
vm.min_free_kbytes = 32768
vm.mmap_min_addr = 4096
vm.nr_pdflush_threads = 0
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_memory = 2
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 100
vm.user_reserve_kbytes = 7914
vm.vfs_cache_pressure = 100

What am I doing wrong (except for using a SOHO router for tor, lol)?

PS: Using DD-WRT firmware

EDIT: Another OOM killer log - this time nothing about resetbutton:

kthreadd invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0
CPU: 0 PID: 2 Comm: kthreadd Tainted: P             3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
 r6:c782fb80 r5:00000000 r4:c7834000 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007c5cc>] (dump_header.isra.13+0x84/0x194)
[<c007c548>] (dump_header.isra.13+0x0/0x194) from [<c007c958>] (oom_kill_process+0x90/0x3e8)
[<c007c8c8>] (oom_kill_process+0x0/0x3e8) from [<c007d17c>] (out_of_memory+0x2c0/0x304)
[<c007cebc>] (out_of_memory+0x0/0x304) from [<c007fcbc>] (__alloc_pages_nodemask+0x5b0/0x694)
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c002bf70>] (copy_process.part.68+0xdc/0xe30)
[<c002be94>] (copy_process.part.68+0x0/0xe30) from [<c002cdd8>] (do_fork+0xe0/0x2bc)
[<c002ccf8>] (do_fork+0x0/0x2bc) from [<c002cfe4>] (kernel_thread+0x30/0x38)
 r9:00000001 r8:00000000 r7:c7834000 r6:c0454014 r5:c04096a8
r4:c1109e88
[<c002cfb4>] (kernel_thread+0x0/0x38) from [<c004be50>] (kthreadd+0xe4/0x12c)
[<c004bd6c>] (kthreadd+0x0/0x12c) from [<c0008c38>] (ret_from_fork+0x14/0x3c)
Mem-info:
Normal per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
HighMem per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
active_anon:7196 inactive_anon:7223 isolated_anon:0
 active_file:1054 inactive_file:1422 isolated_file:0
 unevictable:199 dirty:3 writeback:0 unstable:0
 free:22911 slab_reclaimable:969 slab_unreclaimable:6549
 mapped:685 shmem:0 pagetables:210 bounce:0
 free_cma:0
Normal free:29372kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:44kB inactive_file:128kB unevictable:44kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3876kB slab_unreclaimable:26196kB kernel_stack:576kB pagetables:840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1064 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:62272kB min:128kB low:8772kB high:17416kB active_anon:28784kB inactive_anon:28892kB active_file:4068kB inactive_file:5676kB unevictable:752kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:12kB writeback:0kB mapped:2740kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 465*4kB (UEM) 285*8kB (UEM) 219*16kB (UEM) 92*32kB (UEM) 51*64kB (UEM) 25*128kB (UEM) 12*256kB (UEM) 2*512kB (U) 0*1024kB 0*2048kB 2*4096kB (R) = 29340kB
HighMem: 5997*4kB (UM) 2743*8kB (UM) 478*16kB (UM) 27*32kB (UM) 2*64kB (M) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 1*4096kB (R) = 62252kB
5489 total pagecache pages
2805 pages in swap cache
Swap cache stats: add 1237151, delete 1234346, find 539424/688368
Free swap  = 484712kB
Total swap = 524284kB
65536 pages of RAM
23095 free pages
1661 reserved pages
7517 slab pages
557492 pages shared
2805 pages swap cached
[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  622]     0   622      189        1       3       12             0 hotplug2
[  626]     0   626      199       31       3        6             0 mstpd
[  971]     0   971      217        1       3        7             0 dropbear
[  987]     0   987      297      105       3       30             0 dnsmasq
[ 1112]     0  1112      315       61       4       10             0 ttraff
[ 1195]     0  1195      209       40       3        2             0 dhcp6c
[ 1288]     0  1288      755        1       4       82             0 httpd
[ 1330]     0  1330      315       38       3       17             0 process_monitor
[ 1334]     0  1334      366       89       3       11             0 upnp
[ 1337]     0  1337      334       50       4        9             0 wland
[ 1338]     0  1338      190       45       3        4             0 igmprt
[ 1339]     0  1339      291       30       3        5             0 udhcpc
[ 1449]     0  1449      182       32       3        2             0 cron
[ 1457]     0  1457      370       36       3       32             0 nas
[ 1460]     0  1460      370       34       4       32             0 nas
[ 1495]     0  1495      291       20       3        5             0 syslogd
[ 1497]     0  1497      291       37       3        2             0 klogd
[ 1500]     0  1500      334       32       3       12             0 resetbutton
[ 1684] 65534  1684     1008       15       5      499             0 tinyproxy
[ 1690] 65534  1690     1021        5       5      507             0 tinyproxy
[ 1691] 65534  1691     1021        5       5      507             0 tinyproxy
[ 1692] 65534  1692     1021        5       5      507             0 tinyproxy
[ 1693] 65534  1693     1008        4       5      501             0 tinyproxy
[ 1694] 65534  1694     1008        4       5      501             0 tinyproxy
[ 1695] 65534  1695     1008        4       5      501             0 tinyproxy
[ 1696] 65534  1696     1008        4       5      501             0 tinyproxy
[ 1697] 65534  1697     1008        4       5      501             0 tinyproxy
[ 1698] 65534  1698     1008        4       5      501             0 tinyproxy
[ 1699] 65534  1699     1008        4       5      501             0 tinyproxy
[ 1730]    13  1730      580       10       4       37             0 polipo
[ 1806]  1000  1806     2828       78       5      116             0 wrapper
[ 1921]     0  1921      241       51       3       19             0 dropbear
[ 1923]     0  1923      293        1       3        8             0 sh
[16658]   106 16643    33786    12722      65     7647             0 tor
[16661]     0 16661      804        4       4       85             0 bash
[17143]     0 17143      790      167       4       26             0 top
[20851]     0 20851      320       64       3       12             0 radvd
[20853]     0 20853      320       33       3       17             0 radvd
Out of memory: Kill process 16658 (tor) score 104 or sacrifice child
Killed process 16658 (tor) total-vm:135144kB, anon-rss:49088kB, file-rss:1800kB
tor: page allocation failure: order:0, mode:0x20058
CPU: 1 PID: 16658 Comm: tor Tainted: P             3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
 r6:00000000 r5:00000000 r4:00020058 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007def4>] (warn_alloc_failed+0x100/0x110)
[<c007ddf4>] (warn_alloc_failed+0x0/0x110) from [<c007faf0>] (__alloc_pages_nodemask+0x3e4/0x694)
 r3:00040001 r2:00000000
 r7:c0405180 r6:c6488000 r5:00000000 r4:00020058
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c007a274>] (find_or_create_page+0x48/0x94)
[<c007a22c>] (find_or_create_page+0x0/0x94) from [<c00cfc08>] (__getblk+0x16c/0x308)
 r9:00000000 r8:0220f844 r7:c7418110 r6:00000001 r5:00001000
r4:00000000
[<c00cfa9c>] (__getblk+0x0/0x308) from [<c00d0668>] (__bread+0x1c/0xd8)
[<c00d064c>] (__bread+0x0/0xd8) from [<bf5e96c8>] (_8+0x208/0x4a0 [ext2])
 r4:c6489b58
[<bf5e963c>] (_8+0x17c/0x4a0 [ext2]) from [<bf5ea2e0>] (_35+0x78/0x884 [ext2])
[<bf5ea268>] (_35+0x0/0x884 [ext2]) from [<c00d6da8>] (do_mpage_readpage+0x268/0x6d4)
[<c00d6b40>] (do_mpage_readpage+0x0/0x6d4) from [<c00d72d4>] (mpage_readpages+0xc0/0x104)
[<c00d7214>] (mpage_readpages+0x0/0x104) from [<bf5e99ac>] (_61+0x24/0x28 [ext2])
[<bf5e9988>] (_61+0x0/0x28 [ext2]) from [<c0082f08>] (__do_page_cache_readahead+0x170/0x1f8)
[<c0082d98>] (__do_page_cache_readahead+0x0/0x1f8) from [<c0083428>] (ra_submit+0x30/0x38)
[<c00833f8>] (ra_submit+0x0/0x38) from [<c007bb4c>] (filemap_fault+0x1bc/0x434)
[<c007b990>] (filemap_fault+0x0/0x434) from [<c008e5a4>] (__do_fault+0xac/0x490)
[<c008e4f8>] (__do_fault+0x0/0x490) from [<c0091284>] (handle_pte_fault+0x78/0x6d4)
[<c009120c>] (handle_pte_fault+0x0/0x6d4) from [<c0091968>] (__handle_mm_fault+0x88/0x90)
[<c00918e0>] (__handle_mm_fault+0x0/0x90) from [<c00920e0>] (handle_mm_fault+0x28/0x2c)
[<c00920b8>] (handle_mm_fault+0x0/0x2c) from [<c001948c>] (do_page_fault+0x118/0x288)
[<c0019374>] (do_page_fault+0x0/0x288) from [<c000861c>] (do_PrefetchAbort+0x40/0x9c)
[<c00085dc>] (do_PrefetchAbort+0x0/0x9c) from [<c0008b58>] (ret_from_exception+0x0/0x10)
Exception stack(0xc6489fb0 to 0xc6489ff8)
9fa0:                                     0000021c 00000000 6df1d800 6df1d800
9fc0: b470b880 0000000f da4afeed b6fe04bc 00000040 5eedf00d b6fdb8f4 b6fba134
9fe0: 00000000 b470b858 b6f0ead4 b6f0ead4 60000010 ffffffff
 r8:00000040 r7:b6fe04bc r6:ffffffff r5:60000010 r4:b6f0ead4
Mem-info:
Normal per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
HighMem per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
active_anon:7196 inactive_anon:7223 isolated_anon:0
 active_file:1028 inactive_file:1457 isolated_file:0
 unevictable:199 dirty:3 writeback:0 unstable:0
 free:22844 slab_reclaimable:969 slab_unreclaimable:6547
 mapped:685 shmem:0 pagetables:210 bounce:0
 free_cma:0
Normal free:29092kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:44kB inactive_file:128kB unevictable:44kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3876kB slab_unreclaimable:26188kB kernel_stack:576kB pagetables:840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:62252kB min:128kB low:8772kB high:17416kB active_anon:28784kB inactive_anon:28892kB active_file:4068kB inactive_file:5700kB unevictable:752kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:12kB writeback:0kB mapped:2740kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 467*4kB (UEM) 285*8kB (UEM) 219*16kB (UEM) 93*32kB (UEM) 50*64kB (EM) 25*128kB (UEM) 11*256kB (UEM) 2*512kB (U) 0*1024kB 0*2048kB 2*4096kB (R) = 29060kB
HighMem: 5997*4kB (UM) 2743*8kB (UM) 478*16kB (UM) 27*32kB (UM) 2*64kB (M) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 1*4096kB (R) = 62252kB
5489 total pagecache pages
2805 pages in swap cache
Swap cache stats: add 1237151, delete 1234346, find 539424/688368
Free swap  = 484712kB
Total swap = 524284kB
65536 pages of RAM
23025 free pages
1661 reserved pages
7515 slab pages
819780 pages shared
2805 pages swap cached
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets

And again, Free swap = 484712kB - huge amount not used. Why?

  • Some process called resetbutton caused the OOM. Tor was just the unfortunate victim. Of course I wouldn't use DD-WRT for anything now that they've sold out... – Michael Hampton Jul 28 '15 at 14:37
  • Sure, but why OOM was caused? There was still plenty of swap! – Evengard Jul 28 '15 at 14:48
  • Just for giggles, add this to your tor config: DisableAllSwap 0 – Aaron Aug 5 '15 at 19:36
  • Added, but that is the default, no? – Evengard Aug 5 '15 at 20:51
  • Without knowing your compile options, it may not be. There is a flag to enable expensive hardening options and I believe that changes some defaults, including that one. – Aaron Aug 7 '15 at 20:43
11
+50

Your problem actually lies here:

Normal 
free:29372kB
min:32768kB 
low:40960kB
high:49152kB
active_anon:0kB 
inactive_anon:0kB 
active_file:44kB 
inactive_file:128kB 
unevictable:44kB 
isolated(anon):0kB 
isolated(file):0kB 
present:131072kB 
managed:124196kB 
mlocked:0kB 
dirty:0kB 
writeback:0kB 
mapped:0kB 
shmem:0kB 
slab_reclaimable:3876kB 
slab_unreclaimable:26196kB 
kernel_stack:576kB 
pagetables:840kB 
unstable:0kB 
bounce:0kB 
free_cma:0kB 
writeback_tmp:0kB 
pages_scanned:1064 
all_unreclaimable? no

The memory on your router is split up into zones, one zone -- HighMem contains 128MiB of memory, the other zone, Normal contains 128MiB of memory.

The zone are functionally independent of one another. The memory manager works in each zone in an isolated way.

oom-killer is invoked in the normal zone. You can practically eliminate 128Mib of memory when you are looking at your request.

Now to the actual anomaly:

free: 29372kB
min: 32768kB

The min field is the kernels interpretation of 'when I go below this threshold, I will have to do drastic things to reclaim memory -- like OOM-killer'. Whats highly unusual about this field in your description is just how much is set as the min amount. This value typically floats at about 2% of your memory, in this case its 25% of the entire zone. Its no wonder you oom-kill so aggressively!

Lo and behlod when reviewing your question output the reason for the problem lays here:

vm.min_free_kbytes = 32768

Alter this value to be something way lower. Perhaps 1024 would be a more reasonable number.

You also shows some 'orphaned tcp socket' errors. You may also wish to reduce the values for net.ipv4.tcp_wmem and net.ipv4.tcp_rmem which will permit you handle more sockets (at the cost of latency).

So, to answer your question why are you killing processes with so much free swap - the answer to that is the vm.min_free_kb setting is ridiculously high for the amount of actual memory you have in each zone. Reduce this value from 32MiB to 1MiB.

  • Good catch Matthew on min_free_kbytes. That said I am confused by your explanation: the first failing example uses a gfp_mask of 0x2000d0. Since the __GFP_NOMEMALLOC (0x100000) was not asserted it is my understanding that the malloc() is requesting emergency reserves, which seems sensible for resetbutton to do. Yet you're saying it's specifically vm.min_free_kb causing that reserve allocation to fail. What gives: are userspace programs not allowed to allocate from that reserved space in any circumstance? (I'm looking at permalink.gmane.org/gmane.linux.kernel/436364 for background) – Jesse Scherer Aug 3 '15 at 22:44
  • 1
    @jjjjjoe In example one thats GFP_NOTRACK|GFP_KERNEL. GPT_NOMEMALLOC you refer to is 0x10000, not 0x100000 which is GFP_NOACCOUNT, additionally GFP_HIGH must be set to use emergency reserves. The GFP_NOTRACK thats set turns off a bunch of internal kernel checks for warning about uninitialized memory. Userspace has no direct access to the GFP flags, userspace can only call system calls like mmap(), brk() or shmget(). In this case, oom-killer was called via a slabcache allocation for the names_cache slab produced by resetbutton trying to open a file. – Matthew Ife Aug 3 '15 at 23:40
  • 1
    @jjjjjoe To respond to the question directly, I'm saying the going under the min watermark invokes the OOM-Killer for low order allocations (0, 1, 2 or 3). Do userspace applications have access to emergency pools? No, not directly. Userspace cannot directly manipulate internal GFP flags. This can be done indirectly via a system call which may set the flag in kernel code under certain circumstances. If this interests you see lxr.free-electrons.com/source/include/linux/gfp.h for GFP definitions and meanings. – Matthew Ife Aug 3 '15 at 23:40
  • As I jokingly commented in reply to my own incorrect guess, I guess it's not my day if I can't count zeroes. In all seriousness, thank you for the detailed explanation of what is going on here. – Jesse Scherer Aug 4 '15 at 15:25
  • Thank you! If you don't mind, I will check all of that and then report back. Seems like a good find. But still why it didn't used the swap to allocate the memory? – Evengard Aug 4 '15 at 17:00
3

EDIT: This answer totally misses the problem. See Matthew Ife's post about the crazy high setting on vm.min_free_kbytes

The router and tor have been running for a while, correct?

I suspect you are being hit by external (memory) fragmentation. If you look in your OOM logs you can see that there are many chunks of available memory, but if none of them are as big as what tor is asking for, the malloc() will fail.

Normal: 465*4kB (UEM) 285*8kB (UEM) 219*16kB (UEM) 92*32kB (UEM) 51*64kB (UEM) 25*128kB (UEM) 12*256kB (UEM) 2*512kB (U) 0*1024kB 0*2048kB 2*4096kB (R) = 29340kB
HighMem: 5997*4kB (UM) 2743*8kB (UM) 478*16kB (UM) 27*32kB (UM) 2*64kB (M) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 1*4096kB (R) = 62252kB

Edit: As Matthew points out, the offending malloc()s are asking for order-0 (single page) and order-1 (two page) allocations. I am clearly barking up the wrong tree.

There is a great explanation of what is happening over at elinux.org. Unfortunately the memory compaction which is mentioned in that presentation does not appear to be tunable in DD-WRT, though I can find changesets going back to 2010 which indicate it is in the kernel.

As to what you should do about tor I am less clear. Maybe you can run a watchdog for it, or opportunistically restart the process (which should help the kernel find space to clean up memory). It's also possible that having the swappiness set so high is slowing down the memory compaction process, but I'm speculating at this point.

  • I am afraid this is incorrect. This buddyinfo value is dictated by the order size of the allocation, not the amount being allocated. The highest order in all examples is order 1 (for which plenty of memory exists). – Matthew Ife Aug 3 '15 at 19:58
  • Today is not my day, it seems. – Jesse Scherer Aug 3 '15 at 22:27
  • Thanks for trying to help though – Evengard Aug 4 '15 at 17:26
2

You have a couple of obvious problems.

First:

vm.swappiness = 100

You've told the kernel to swap as aggressively as possible, even when it's not strictly necessary to do so.

On an embedded device (which usually won't have swap at all) this is completely insane. This should be set to 0 (probably best) or 1.

Second:

vm.overcommit_memory = 2

Aside from preventing any overcommit at all, and preventing the system from using all free memory and swap, this also causes additional memory to be reserved for admin and user processes as specified in admin_reserve_kbytes and user_reserve_kbytes.

I can't find a single system I manage, embedded, desktop or server, where this setting is anything other than 0.

Finally, about DD-WRT... I'm not happy with them. They went overly commercial some time back and stopped making regular releases and updates. These days I recommend OpenWrt as a platform for building embedded systems.

  • Seems like you ddidn't understood well my question :) I enabled swap on the device (on external HDD connected by USB), and I want to use it. So swappiness 100 is what I need. Well, the same thing happened (oom killer starting when there is plenty of swap) when overcommit_memory was 0 (the default one) - I was hoping it was overcommit problems. Why the swap isn't used and instead OOM killer was invoked? And well, while there is no open source drivers for AC Wifi module used in that router, OpenWrt don't have a version. dd-wrt seems like the only option for me. – Evengard Jul 28 '15 at 15:05
  • I understood you want to have swap. But you didn't understand the answer. You really do not want to swap aggressively! You want to use your RAM first! – Michael Hampton Jul 28 '15 at 15:07
  • Still that doesn't explain the OOM killer anyway – Evengard Jul 28 '15 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.