0

My system is a software network intercepter, which runs on a Ubuntu 14.04.4 LTS (GNU/Linux 3.16.0-71-generic x86_64). I have it deployed in several sites, yet a specific one gives me the following weird problem. During the system's operation it consumes a lot of memory, until it fills the entire swap up and becomes very low. Mind you, that is not one of our most input intense installations. My problem is that when I open up htop, the amount of memory used by processes doesn't add up. So, I have turned off all of my system's processes. Furthermore, we use MongoDB, which uses up a lot of memory, so I have turned that off as well. After turning off everything, and the CPU goes down to virtually zero, there are still 20GB used, seemingly leaked due to some kernel issue. So I turn to /proc/meminfo, yet the results simply do not tell me anything regarding where the missing memory is:

MemTotal:       32948744 kB
MemFree:        10695472 kB
MemAvailable:   10925256 kB
Buffers:           34780 kB
Cached:           294564 kB
SwapCached:          564 kB
Active:           262064 kB
Inactive:         158464 kB
Active(anon):      82344 kB
Inactive(anon):     8896 kB
Active(file):     179720 kB
Inactive(file):   149568 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      16777180 kB
SwapFree:       16753824 kB
Dirty:             11816 kB
Writeback:             0 kB
AnonPages:         90708 kB
Mapped:            32500 kB
Shmem:                56 kB
Slab:             168804 kB
SReclaimable:     138864 kB
SUnreclaim:        29940 kB
KernelStack:        3040 kB
PageTables:         4600 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    33251552 kB
Committed_AS:     538280 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      203084 kB
VmallocChunk:   34359505748 kB
HardwareCorrupted:     0 kB
AnonHugePages:     18432 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       65472 kB
DirectMap2M:    33488896 kB

According to what I've read online regarding the output of meminfo, the numbers simply don't add up - there is no big consumer of memory.

Additionally, here are the top memory users according to htop:

  840 _statsd    20   0 1181M 22672  4024 S  0.0  0.1  5h54:36 statsd /etc/statsd/localConfig.js
 1302 _graphite  20   0  145M 16044  2164 S  0.0  0.0  4h55:38 /usr/bin/python /usr/bin/carbon-cache --c
  837 _statsd    20   0 1178M 13892  3100 S  0.0  0.0  9:15.03 /usr/bin/nodejs proxy.js /etc/statsd/prox
20639 root       20   0  101M  6216  5244 S  0.0  0.0  0:00.03 sshd: paf [priv]
20820 root       20   0  101M  6196  5228 S  0.0  0.0  0:00.02 sshd: paf [priv]
20870 paf        20   0 22504  5208  3268 S  0.0  0.0  0:00.20 -bash
20732 paf        20   0 22408  4860  3020 S  0.0  0.0  0:00.25 -bash
20869 paf        20   0  101M  3692  2732 S  0.0  0.0  0:00.01 sshd: paf@pts/3
20731 paf        20   0  101M  3636  2672 S  0.0  0.0  0:00.12 sshd: paf@pts/1
21101 paf        20   0 26060  3300  2404 R  1.4  0.0  0:00.41 htop
    1 root       20   0 33976  2744  1292 S  0.0  0.0  0:34.74 /sbin/init

Edit (extra diagnostics):

~$ free -m
             total       used       free     shared    buffers     cached
Mem:         32176      23619       8556          0        130        838
-/+ buffers/cache:      22650       9526
Swap:        16383         37      16346

slabtop:

 Active / Total Objects (% used)    : 631869 / 707961 (89.3%)
 Active / Total Slabs (% used)      : 23836 / 23836 (100.0%)
 Active / Total Caches (% used)     : 70 / 102 (68.6%)
 Active / Total Size (% used)       : 174386.89K / 194368.76K (89.7%)
 Minimum / Average / Maximum Object : 0.01K / 0.27K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
221169 198528  89%    0.10K   5671       39     22684K buffer_head
123690 121130  97%    0.19K   5890       21     23560K dentry
 82768  81847  98%    0.98K   5173       16     82768K ext4_inode_cache
 50708  31368  61%    0.57K   1811       28     28976K radix_tree_node
 39372  30686  77%    0.04K    386      102      1544K ext4_extent_status
 31790  31790 100%    0.12K    935       34      3740K kernfs_node_cache
 14091   6729  47%    0.19K    671       21      2684K kmalloc-192
 13376   8603  64%    0.06K    209       64       836K kmalloc-64
 13312  11934  89%    0.03K    104      128       416K kmalloc-32
 11776  11776 100%    0.03K     92      128       368K jbd2_revoke_record_s
  8500   7085  83%    0.05K    100       85       400K shared_policy_node
  8064   8064 100%    0.07K    144       56       576K Acpi-ParseExt
  7700   7700 100%    0.57K    275       28      4400K inode_cache
  7680   7680 100%    0.01K     15      512        60K kmalloc-8
  7424   7424 100%    0.02K     29      256       116K kmalloc-16
  6928   3739  53%    0.25K    433       16      1732K kmalloc-256
  5712   5265  92%    0.08K    112       51       448K anon_vma
  5088   4025  79%    0.12K    159       32       636K kmalloc-128
  4992   4992 100%    0.06K     78       64       312K ext4_free_data
  4312   4312 100%    0.07K     77       56       308K ext4_io_end
  3948   3948 100%    0.09K     94       42       376K kmalloc-96
  3876   3876 100%    0.04K     38      102       152K Acpi-Namespace
  3575   2475  69%    0.63K    143       25      2288K proc_inode_cache
  2232   2124  95%    0.11K     62       36       248K jbd2_journal_head
  2016   2016 100%    0.14K     72       28       288K ext4_groupinfo_4k
  1888   1459  77%    0.50K    118       16       944K kmalloc-512
  1804   1804 100%    0.36K     82       22       656K blkdev_requests
  1575   1479  93%    0.31K     63       25       504K nf_conntrack_ffffffff81cd3800
  1512   1512 100%    0.11K     42       36       168K task_delay_info
  1400   1400 100%    0.62K     56       25       896K sock_inode_cache
  1232   1158  93%    1.00K     77       16      1232K kmalloc-1024
  1104   1104 100%    0.09K     24       46        96K ftrace_event_file
  1104   1048  94%    0.66K     46       24       736K shmem_inode_cache
  1020   1020 100%    0.02K      6      170        24K numa_policy
   903    903 100%    0.38K     43       21       344K scsi_cmd_cache
   896    850  94%    2.00K     56       16      1792K kmalloc-2048
   819    819 100%    0.10K     21       39        84K blkdev_ioc
   810    810 100%    1.06K     27       30       864K UDPv6
   732    403  55%    2.55K     61       12      1952K task_struct
   560    544  97%    0.25K     35       16       140K dquot
   510    510 100%    0.08K     10       51        40K Acpi-State
   510    510 100%    0.05K      6       85        24K jbd2_journal_handle
   504    504 100%    1.75K     28       18       896K TCP
   476    476 100%    1.12K     17       28       544K signal_cache
   368    368 100%    0.25K     23       16        92K tw_sock_TCP
   320    320 100%    1.94K     20       16       640K TCPv6
   315    315 100%    2.05K     21       15       672K idr_layer_cache
   315    276  87%    2.06K     21       15       672K sighand_cache
   288    288 100%    0.25K     18       16        72K tw_sock_TCPv6
   256    256 100%    0.06K      4       64        16K kmem_cache_node
   256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s
   234    234 100%    0.88K     13       18       208K mm_struct
   225    225 100%    0.62K      9       25       144K files_cache
   219    219 100%    0.05K      3       73        12K ip_fib_trie
   204    204 100%    0.94K     12       17       192K UNIX
   192    168  87%    4.00K     24        8       768K kmalloc-4096
   192    192 100%    0.12K      6       32        24K ext4_allocation_context
   150    150 100%    0.16K      6       25        24K sigqueue

Could anyone please give me a lead regarding how to investigate this issue further?

  • Post the output of free -m. – EEAA Jun 15 '16 at 7:30
  • 1
    Maybe the output of slabtop too, perhaps compared with a machine with similar hardware but without the issue if you have that. – Mat Jun 15 '16 at 9:27
  • Edited into original post. – borisdan Jun 15 '16 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.