Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Sometimes I find that 100% of my small Debian machine's RAM is used, but I can't figure out how to find this leak: top show that all memory is used, but I can't find any process that consumed this memory in process listing (processes sorted by field 'n' - %MEM):

top - 12:50:41 up 8 days,  1:13,  1 user,  load average: 0.06, 0.15, 1.63
Tasks: 101 total,   1 running, 100 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1016252k total,   996428k used,    19824k free,      684k buffers
Swap:  1983480k total,   105052k used,  1878428k free,    16884k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                             
13946 www-data  20   0  180m  39m 2588 S    0  3.9   0:08.87 php5-fpm                            
14143 www-data  20   0  161m  30m 3276 S    0  3.0   0:00.46 php5-fpm                            
14142 www-data  20   0  157m  29m 3012 S    0  3.0   0:01.83 php5-fpm                            
1567 mysql     20   0  296m 7740 2344 S    0  0.8   4:37.29 mysqld                              
14197 root      20   0 19072 1320  996 R    0  0.1   0:01.44 top                                 
1110 nginx     20   0 30952  940  548 S    0  0.1   0:17.17 nginx                               
14091 root      20   0 19272  904  660 S    0  0.1   0:00.06 bash                                
13968 root      20   0  115m  856  628 S    0  0.1   0:00.09 console-kit-dae                     
1077 root      20   0  117m  744  484 S    0  0.1   7:46.09 rsyslogd                            
1100 messageb  20   0 23524  488  236 S    0  0.0  89:56.14 dbus-daemon     nsole-kit-dae                     
1077 root      20   0  117m  744  484 S    0  0.1   7:46.09 rsyslogd                 

Should I look at VIRT field instead of %MEM?


UPDATED 10 March: Now I think problem appeared again, but I don't experience any lags. Here is what I can get:

# free -m
             total       used       free     shared    buffers     cached
Mem:           992        968         23          0        195        453
-/+ buffers/cache:        320        671
Swap:         1936          0       1936


# ps axl
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
4     0     1     0  20   0   8356   836 -      Ss   ?          0:03 init [2]
1     0     2     0  20   0      0     0 -      S    ?          0:00 [kthreadd]
1     0     3     2 -100  -      0     0 -      S    ?          0:00 [migration/0]
1     0     4     2  20   0      0     0 -      S    ?          0:00 [ksoftirqd/0]
5     0     5     2 -100  -      0     0 -      S    ?          0:00 [watchdog/0]
1     0     6     2 -100  -      0     0 -      S    ?          0:00 [migration/1]
1     0     7     2  20   0      0     0 -      S    ?          0:00 [ksoftirqd/1]
5     0     8     2 -100  -      0     0 -      S    ?          0:00 [watchdog/1]
1     0     9     2  20   0      0     0 -      S    ?          0:48 [events/0]
1     0    10     2  20   0      0     0 -      S    ?          0:05 [events/1]
1     0    11     2  20   0      0     0 -      S    ?          0:00 [cpuset]
1     0    12     2  20   0      0     0 -      S    ?          0:00 [khelper]
1     0    13     2  20   0      0     0 -      S    ?          0:00 [netns]
1     0    14     2  20   0      0     0 -      S    ?          0:00 [async/mgr]
1     0    15     2  20   0      0     0 -      S    ?          0:00 [pm]
1     0    16     2  20   0      0     0 -      S    ?          0:00 [sync_supers]
1     0    17     2  20   0      0     0 -      S    ?          0:00 [bdi-default]
1     0    18     2  20   0      0     0 -      S    ?          0:00 [kintegrityd/0]
1     0    19     2  20   0      0     0 -      S    ?          0:00 [kintegrityd/1]
1     0    20     2  20   0      0     0 -      S    ?          0:00 [kblockd/0]
1     0    21     2  20   0      0     0 -      S    ?          0:00 [kblockd/1]
1     0    22     2  20   0      0     0 -      S    ?          0:00 [kacpid]
1     0    23     2  20   0      0     0 -      S    ?          0:00 [kacpi_notify]
1     0    24     2  20   0      0     0 -      S    ?          0:00 [kacpi_hotplug]
1     0    25     2  20   0      0     0 -      S    ?          0:00 [kseriod]
1     0    28     2  20   0      0     0 -      S    ?          0:07 [kondemand/0]
1     0    29     2  20   0      0     0 -      S    ?          0:05 [kondemand/1]
1     0    30     2  20   0      0     0 -      S    ?          0:00 [khungtaskd]
1     0    31     2  20   0      0     0 -      S    ?          0:00 [kswapd0]
1     0    32     2  25   5      0     0 -      SN   ?          0:00 [ksmd]
1     0    33     2  20   0      0     0 -      S    ?          0:00 [aio/0]
1     0    34     2  20   0      0     0 -      S    ?          0:00 [aio/1]
1     0    35     2  20   0      0     0 -      S    ?          0:00 [crypto/0]
1     0    36     2  20   0      0     0 -      S    ?          0:00 [crypto/1]
1     0   226     2  20   0      0     0 -      S    ?          0:00 [ksuspend_usbd]
1     0   227     2  20   0      0     0 -      S    ?          0:00 [khubd]
1     0   228     2  20   0      0     0 -      S    ?          0:00 [ata/0]
1     0   229     2  20   0      0     0 -      S    ?          0:00 [ata/1]
1     0   230     2  20   0      0     0 -      S    ?          0:00 [ata_aux]
1     0   232     2  20   0      0     0 -      S    ?          0:00 [scsi_eh_0]
1     0   233     2  20   0      0     0 -      S    ?          0:00 [scsi_eh_1]
1     0   235     2  20   0      0     0 -      S    ?          0:00 [scsi_eh_2]
1     0   238     2  20   0      0     0 -      S    ?          0:00 [scsi_eh_3]
1     0   279     2  20   0      0     0 -      S    ?          0:04 [kjournald]
5     0   357     1  16  -4  17160  1172 -      S<s  ?          0:00 udevd --daemon
1     0   562     2  20   0      0     0 -      S    ?          0:00 [i915]
1     0   579     2  20   0      0     0 -      S    ?          0:02 [flush-8:0]
1     0   580     2  20   0      0     0 -      S    ?          0:00 [hd-audio0]
5     1   844     1  20   0   8096   536 -      Ss   ?          0:00 /sbin/portmap
5   103   856     1  20   0  14384   892 -      Ss   ?          0:00 /sbin/rpc.statd
5     0  1066     1  20   0 119964  1920 -      Sl   ?          0:01 /usr/sbin/rsyslogd -c4
5   101  1097     1  20   0  23524  1116 -      Ss   ?          0:13 /usr/bin/dbus-daemon --syste
5     0  1106     1  20   0  30404  1292 -      Ss   ?          0:00 nginx: master process /usr/s
5   110  1107  1106  20   0  30952  3164 -      S    ?          0:03 nginx: worker process
5     0  1108     1  20   0  78448  4044 -      Ssl  ?          0:01 /usr/sbin/NetworkManager
4     0  1114     1  20   0  59892  2536 -      S    ?          0:00 /usr/sbin/modem-manager
4     0  1124     1  20   0  28264  2148 -      S    ?          0:00 /sbin/wpa_supplicant -u -s
5     0  1138     1  20   0 136060  4796 -      Ss   ?          0:05 php-fpm: master process (/et
1     0  1146     1  20   0   3920   648 -      Ss   ?          0:00 /usr/sbin/acpid
1     1  1166     1  20   0  18716   444 -      Ss   ?          0:00 /usr/sbin/atd
4     0  1187  1108  20   0   6756  1392 -      S    ?          0:00 /sbin/dhclient -d -4 -sf /us
5   104  1193     1  20   0  33892  1664 -      S    ?          0:01 avahi-daemon: running [debia
1   104  1194  1193  20   0  33764   532 -      S    ?          0:00 avahi-daemon: chroot helper
5     0  1210     1  20   0  29308  1948 -      Ss   ?          0:00 /usr/sbin/bluetoothd
1     0  1220     2  20   0      0     0 -      S    ?          0:00 [bluetooth]
1     0  1298     1  20   0  22428  1080 -      Ss   ?          0:00 /usr/sbin/cron
5     0  1318   357  18  -2  17156  1076 -      S<   ?          0:00 udevd --daemon
5     0  1319   357  18  -2  17156  1076 -      S<   ?          0:00 udevd --daemon
4     0  1325     1  20   0  75000  2724 -      Ss   ?          0:00 /usr/sbin/cupsd -C /etc/cups
1     0  1345     1  20   0  62684  1240 -      Ss   ?          0:38 /usr/sbin/kerneloops
1     0  1368     2  20   0      0     0 -      S    ?          0:00 [kconservative/0]
1     0  1369     2  20   0      0     0 -      S    ?          0:00 [kconservative/1]
4 65534  1421     1  20   0  62700  1244 -      Sl   ?          0:04 /usr/bin/memcached -m 64 -p
4     0  1472     1  20   0   3956   644 -      S    ?          0:00 /bin/sh /usr/bin/mysqld_safe
5     0  1604     1  20   0  49176  1140 -      Ss   ?          0:00 /usr/sbin/sshd
4   112  1611  1472  20   0 237876 34476 -      Sl   ?          1:25 /usr/sbin/mysqld --basedir=/
0     0  1612  1472  20   0   3856   664 -      S    ?          0:00 logger -t mysqld -p daemon.e
0   111  1675     1  20   0 102296  6816 -      S    ?          0:01 /usr/lib/postgresql/8.4/bin/
1   111  1861  1675  20   0 102296  1744 -      Ss   ?          0:13 postgres: writer process
1   111  1862  1675  20   0 102296  1536 -      Ss   ?          0:09 postgres: wal writer process
1   111  1863  1675  20   0 102296  1696 -      Ss   ?          0:04 postgres: autovacuum launche
1   111  1864  1675  20   0  73704  1412 -      Ss   ?          0:02 postgres: stats collector pr
5   102  2293     1  20   0  48252  1100 -      Ss   ?          0:00 /usr/sbin/exim4 -bd -q30m
0     0  2314     1  20   0   5932   624 -      Ss+  tty1       0:00 /sbin/getty 38400 tty1
0     0  2315     1  20   0   5932   628 -      Ss+  tty2       0:00 /sbin/getty 38400 tty2
0     0  2316     1  20   0   5932   628 -      Ss+  tty3       0:00 /sbin/getty 38400 tty3
0     0  2317     1  20   0   5932   624 -      Ss+  tty4       0:00 /sbin/getty 38400 tty4
0     0  2318     1  20   0   5932   628 -      Ss+  tty5       0:00 /sbin/getty 38400 tty5
0     0  2319     1  20   0   5932   628 -      Ss+  tty6       0:00 /sbin/getty 38400 tty6
4     0 14409     1  20   0 120560  3848 -      Sl   ?          0:01 /usr/sbin/console-kit-daemon
5    33 16932  1138  20   0 197548 63748 -      S    ?          3:14 php-fpm: pool www
5    33 19727  1138  20   0 189540 58428 -      S    ?          2:10 php-fpm: pool www
5    33 24159  1138  20   0 177920 45164 -      S    ?          1:00 php-fpm: pool www
4     0 26071     1  10 -10  13404  3172 -      S<L  ?          0:01 /usr/bin/atop -a -w /var/log
4     0 26743  1604  20   0  83124  3584 -      Ss   ?          0:00 sshd: debian [priv]
5  1000 26749 26743  20   0  83124  1644 -      S    ?          0:00 sshd: debian@pts/0
0  1000 26750 26749  20   0  25384  8224 -      Ss   pts/0      0:00 -bash
4     0 26783 26750  20   0  49632  1688 -      S    pts/0      0:00 su
0     0 26790 26783  20   0  19284  2048 -      S    pts/0      0:00 bash
4     0 26798 26790  20   0   7924   964 -      R+   pts/0      0:00 ps axl

Updated 18 April: This time I definitely got that problem again. Here is what I got:

root@debian:/home/debian# free -m
             total       used       free     shared    buffers     cached
Mem:           992        927         65          0          1         25
-/+ buffers/cache:        900         91
Swap:         1936        185       1751

root@debian:/home/debian# ps axl
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
4     0     1     0  20   0   8356   652 -      Ss   ?          0:04 init [2]  
1     0     2     0  20   0      0     0 -      S    ?          0:00 [kthreadd]
1     0     3     2 -100  -      0     0 -      S    ?          0:11 [migration/0]
1     0     4     2  20   0      0     0 -      S    ?          0:04 [ksoftirqd/0]
5     0     5     2 -100  -      0     0 -      S    ?          0:00 [watchdog/0]
1     0     6     2 -100  -      0     0 -      S    ?          0:11 [migration/1]
1     0     7     2  20   0      0     0 -      S    ?          0:06 [ksoftirqd/1]
5     0     8     2 -100  -      0     0 -      S    ?          0:00 [watchdog/1]
1     0     9     2  20   0      0     0 -      S    ?         28:27 [events/0]
1     0    10     2  20   0      0     0 -      S    ?         27:31 [events/1]
1     0    11     2  20   0      0     0 -      S    ?          0:00 [cpuset]
1     0    12     2  20   0      0     0 -      S    ?          0:00 [khelper]
1     0    13     2  20   0      0     0 -      S    ?          0:00 [netns]
1     0    14     2  20   0      0     0 -      S    ?          0:00 [async/mgr]
1     0    15     2  20   0      0     0 -      S    ?          0:00 [pm]
1     0    16     2  20   0      0     0 -      S    ?          0:00 [sync_supers]
1     0    17     2  20   0      0     0 -      S    ?          0:00 [bdi-default]
1     0    18     2  20   0      0     0 -      S    ?          0:00 [kintegrityd/0]
1     0    19     2  20   0      0     0 -      S    ?          0:00 [kintegrityd/1]
1     0    20     2  20   0      0     0 -      S    ?          0:01 [kblockd/0]
1     0    21     2  20   0      0     0 -      S    ?          0:01 [kblockd/1]
1     0    22     2  20   0      0     0 -      S    ?          0:00 [kacpid]
1     0    23     2  20   0      0     0 -      S    ?          0:00 [kacpi_notify]
1     0    24     2  20   0      0     0 -      S    ?          0:00 [kacpi_hotplug]
1     0    25     2  20   0      0     0 -      S    ?          0:00 [kseriod]
1     0    28     2  20   0      0     0 -      S    ?          0:28 [kondemand/0]
1     0    29     2  20   0      0     0 -      S    ?          0:24 [kondemand/1]
1     0    30     2  20   0      0     0 -      S    ?          0:00 [khungtaskd]
1     0    31     2  20   0      0     0 -      S    ?          0:10 [kswapd0]
1     0    32     2  25   5      0     0 -      SN   ?          0:00 [ksmd]
1     0    33     2  20   0      0     0 -      S    ?          0:00 [aio/0]
1     0    34     2  20   0      0     0 -      S    ?          0:00 [aio/1]
1     0    35     2  20   0      0     0 -      S    ?          0:00 [crypto/0]
1     0    36     2  20   0      0     0 -      S    ?          0:00 [crypto/1]
1     0   198     2  20   0      0     0 -      S    ?          0:00 [ksuspend_usbd]
1     0   213     2  20   0      0     0 -      S    ?          0:00 [khubd]
1     0   224     2  20   0      0     0 -      S    ?          0:00 [ata/0]
1     0   225     2  20   0      0     0 -      S    ?          0:00 [ata/1]
1     0   226     2  20   0      0     0 -      S    ?          0:00 [ata_aux]
1     0   229     2  20   0      0     0 -      S    ?          0:00 [scsi_eh_0]
1     0   231     2  20   0      0     0 -      S    ?          0:00 [scsi_eh_1]
1     0   233     2  20   0      0     0 -      S    ?          0:00 [scsi_eh_2]
1     0   234     2  20   0      0     0 -      S    ?          0:00 [scsi_eh_3]
1     0   275     2  20   0      0     0 -      S    ?          0:06 [flush-8:0]
1     0   276     2  20   0      0     0 -      S    ?          0:36 [kjournald]
5     0   323     1  16  -4  17160   284 -      S<s  ?          0:00 udevd --daemon
1     0   550     2  20   0      0     0 -      S    ?          0:00 [i915]
1     0   565     2  20   0      0     0 -      S    ?          0:00 [hd-audio0]
5     1   841     1  20   0   8096   396 -      Ss   ?          0:00 /sbin/portmap
5   103   853     1  20   0  14384   620 -      Ss   ?          0:00 /sbin/rpc.statd
5    33   869  1100  20   0 177280 39848 -      S    ?          3:34 php-fpm: pool www                                         
5     0  1041     1  20   0 119964  1144 -      Sl   ?          7:03 /usr/sbin/rsyslogd -c4
5     0  1069     1  20   0  30404   248 -      Ss   ?          0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
5   110  1070  1069  20   0  30952  1772 -      S    ?          0:56 nginx: worker process                   
5   101  1072     1  20   0  23524   880 -      Ss   ?         85:03 /usr/bin/dbus-daemon --system
5     0  1081     1  20   0  78448  1440 -      Ssl  ?          6:43 /usr/sbin/NetworkManager
4     0  1087     1  20   0  59892  1220 -      S    ?          5:19 /usr/sbin/modem-manager
4     0  1095     1  20   0  28264   572 -      S    ?          0:00 /sbin/wpa_supplicant -u -s
5     0  1100     1  20   0 136060   700 -      Ss   ?          0:10 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)      
4     0  1106     1  20   0 116052  1404 -      Sl   ?         84:07 /usr/sbin/gdm3
1     0  1119     1  20   0   3920   480 -      Ss   ?          3:05 /usr/sbin/acpid
1     1  1139     1  20   0  18716   268 -      Ss   ?          0:00 /usr/sbin/atd
4     0  1153  1081  20   0   6756   612 -      S    ?          0:00 /sbin/dhclient -d -4 -sf /usr/lib/NetworkManager/nm-dhcp-client.action -pf /var/run/dhclient-eth1.pid -lf /var/lib/dhcp/dhclient-9a8e5759-d3d3-4974-9379-99c403a424e7-eth1.lease -cf /var/run/nm-dhclient-eth1.conf eth1
5   104  1166     1  20   0  33896   924 -      S    ?          3:53 avahi-daemon: running [debian.local]
1   104  1167  1166  20   0  33764   164 -      S    ?          0:00 avahi-daemon: chroot helper
5     0  1181     1  20   0  29308   588 -      Ss   ?          0:00 /usr/sbin/bluetoothd
1     0  1191     2  20   0      0     0 -      S    ?          0:00 [bluetooth]
1     0  1310     1  20   0  62684   540 -      Ss   ?          0:57 /usr/sbin/kerneloops
1     0  1312     1  20   0  22428   748 -      Ss   ?          0:00 /usr/sbin/cron
1     0  1320     2  20   0      0     0 -      S    ?          0:00 [kconservative/0]
1     0  1321     2  20   0      0     0 -      S    ?          0:00 [kconservative/1]
5     0  1328   323  18  -2  17156   220 -      S<   ?          0:00 udevd --daemon
5     0  1332   323  18  -2  17156   220 -      S<   ?          0:00 udevd --daemon
4 65534  1353     1  20   0 128236   660 -      Sl   ?          0:09 /usr/bin/memcached -m 64 -p 11211 -u nobody -l 127.0.0.1
4     0  1365     1  20   0  75000   696 -      Ss   ?          0:00 /usr/sbin/cupsd -C /etc/cups/cupsd.conf
4     0  1428     1  20   0   3956   436 -      S    ?          0:00 /bin/sh /usr/bin/mysqld_safe
4   112  1560  1428  20   0 238616  8120 -      Sl   ?          3:57 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
0     0  1561  1428  20   0   3856   560 -      S    ?          0:00 logger -t mysqld -p daemon.error
5     0  1615     1  20   0  49176   364 -      Ss   ?          0:01 /usr/sbin/sshd
0   111  1665     1  20   0 102304   828 -      S    ?          0:01 /usr/lib/postgresql/8.4/bin/postgres -D /var/lib/postgresql/8.4/main -c config_file=/etc/postgresql/8.4/main/postgresql.conf
1   111  1858  1665  20   0 102304   360 -      Ss   ?          0:35 postgres: writer process                                                                                                    
1   111  1859  1665  20   0 102304   272 -      Ss   ?          0:26 postgres: wal writer process                                                                                                
1   111  1860  1665  20   0 102304   504 -      Ss   ?          0:07 postgres: autovacuum launcher process                                                                                       
1   111  1861  1665  20   0  73712   308 -      Ss   ?          0:05 postgres: stats collector process                                                                                           
5   102  2289     1  20   0  48252   480 -      Ss   ?          0:00 /usr/sbin/exim4 -bd -q30m
0     0  2311     1  20   0   5932   508 -      Ss+  tty1       0:00 /sbin/getty 38400 tty1
0     0  2312     1  20   0   5932   508 -      Ss+  tty2       0:00 /sbin/getty 38400 tty2
0     0  2313     1  20   0   5932   508 -      Ss+  tty3       0:00 /sbin/getty 38400 tty3
0     0  2314     1  20   0   5932   508 -      Ss+  tty4       0:00 /sbin/getty 38400 tty4
0     0  2315     1  20   0   5932   508 -      Ss+  tty5       0:00 /sbin/getty 38400 tty5
0     0  2316     1  20   0   5932   508 -      Ss+  tty6       0:00 /sbin/getty 38400 tty6
4     0  3336     1  10 -10  13404  3172 -      S<L  ?          0:01 /usr/bin/atop -a -w /var/log/atop.log 600
5    33  4165  1100  20   0 186456 30024 -      D    ?          0:01 php-fpm: pool www                                         
5    33  4166  1100  20   0 181048 22824 -      S    ?          0:02 php-fpm: pool www                                         
4     0  4167  1615  20   0  83124   832 -      Ss   ?          0:00 sshd: debian [priv]
5  1000  4171  4167  20   0  83124   628 -      S    ?          0:00 sshd: debian@pts/0
0  1000  4172  4171  20   0  25392  1812 -      Ss   pts/0      0:00 -bash
4     0  4240  4172  20   0  49632  1688 -      S    pts/0      0:00 su
0     0  4247  4240  20   0  19292  2056 -      S    pts/0      0:00 bash
4     0  4255  4247  20   0   7924   960 -      R+   pts/0      0:00 ps axl
4     0 17560     1  20   0 127620  2104 -      Sl   ?         52:46 /usr/sbin/console-kit-daemon --no-daemon

root@debian:/home/debian# cat /proc/meminfo
MemTotal:        1016252 kB
MemFree:           16224 kB
Buffers:            3060 kB
Cached:            30304 kB
SwapCached:        18444 kB
Active:            83036 kB
Inactive:          98796 kB
Active(anon):      67632 kB
Inactive(anon):    83520 kB
Active(file):      15404 kB
Inactive(file):    15276 kB
Unevictable:        3168 kB
Mlocked:            3168 kB
SwapTotal:       1983480 kB
SwapFree:        1838296 kB
Dirty:                40 kB
Writeback:             0 kB
AnonPages:        140328 kB
Mapped:            12564 kB
Shmem:               188 kB
Slab:             787516 kB
SReclaimable:       6520 kB
SUnreclaim:       780996 kB
KernelStack:        1504 kB
PageTables:         5608 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2491604 kB
Committed_AS:     500444 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      539160 kB
VmallocChunk:   34359190284 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       10112 kB
DirectMap2M:     1028096 kB


root@debian:/home/debian# slabtop -osc 
 Active / Total Objects (% used)    : 2790263 / 2795089 (99.8%)
 Active / Total Slabs (% used)      : 58033 / 58033 (100.0%)
 Active / Total Caches (% used)     : 57 / 71 (80.3%)
 Active / Total Size (% used)       : 783632.88K / 785577.16K (99.8%)
 Minimum / Average / Maximum Object : 0.01K / 0.28K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
1369882 1369871  99%    0.53K  45675       30    730800K idr_layer_cache        
1371136 1371134  99%    0.03K  10712      128     42848K kmalloc-32             
  4930   3009  61%    0.55K    170       29      2720K radix_tree_node        
  3015   2900  96%    0.79K    153       20      2448K ext3_inode_cache       
   345    344  99%    4.00K     58        8      1856K kmalloc-4096           
  5208   4217  80%    0.19K    248       21       992K dentry                 
  4158   4141  99%    0.18K    189       22       756K vm_area_struct         
  1215    799  65%    0.59K     45       27       720K inode_cache            
  8823   8818  99%    0.08K    173       51       692K sysfs_dir_cache        
   620    620 100%    0.79K     31       20       496K shmem_inode_cache      
   750    747  99%    0.63K     30       25       480K proc_inode_cache       
   464    445  95%    1.00K     29       16       464K kmalloc-1024           
  2352   2066  87%    0.19K    112       21       448K kmalloc-192            
   208    202  97%    2.00K     13       16       416K kmalloc-2048           
   234    210  89%    1.77K     13       18       416K task_struct            
   688    659  95%    0.50K     43       16       344K kmalloc-512            
   135    114  84%    2.06K      9       15       288K sighand_cache          
  4352   3933  90%    0.06K     68       64       272K kmalloc-64             
  2196   2192  99%    0.11K     61       36       244K buffer_head            
    20     18  90%    8.00K      5        4       160K kmalloc-8192           
   207    177  85%    0.69K      9       23       144K sock_inode_cache       
  1400   1390  99%    0.07K     25       56       100K Acpi-ParseExt          
   384    367  95%    0.25K     24       16        96K kmalloc-256            
    57     42  73%    1.62K      3       19        96K TCP                    
   736    713  96%    0.12K     23       32        92K kmalloc-128            
   115     75  65%    0.69K      5       23        80K files_cache            
    90     75  83%    0.88K      5       18        80K mm_struct              
   105     96  91%    0.75K      5       21        80K RAW                    
    28     28 100%    2.20K      2       14        64K blkdev_queue           
    34     34 100%    1.81K      2       17        64K TCPv6                  
  3072   3063  99%    0.02K     12      256        48K kmalloc-16             
   462    402  87%    0.09K     11       42        44K kmalloc-96             
  1280   1038  81%    0.03K     10      128        40K anon_vma               
  4096   4095  99%    0.01K      8      512        32K kmalloc-8              
    38     38 100%    0.81K      2       19        32K bdev_cache             
     7      7 100%    4.12K      1        7        32K bip-256                
    38     38 100%    0.81K      2       19        32K UDP                    
    32     32 100%    1.00K      2       16        32K UDPv6                  
   252    243  96%    0.11K      7       36        28K task_delay_info  

Anybody can help?

share|improve this question
    
Sow us the output of free -m –  Iain Mar 3 '13 at 7:22
    
lain, sorry, but I've already rebooted machine. now free -m shows Mem: 992 307 685 0 16 103 -/+ buffers/cache: 187 805 Swap: 1936 0 1936 –  muzhig Mar 3 '13 at 8:13
    
VIRT is meaningless. You want to look at either %MEM or RES. Do you have an actual performance issue or is this just cosmetic? –  David Schwartz Mar 3 '13 at 8:28
1  
Typically "can't log in/takes a long time" is just CPU overload, not memory shortage. –  vonbrand Mar 3 '13 at 20:24
1  
You ran out of memory the first time, but not the second time (10th March). –  Michael Hampton Mar 9 '13 at 10:53
show 9 more comments

1 Answer

Looks like a buggy kernel module as slabtop uses up a lot of memory. Something is probably leaking memory in the kernel.

idr_layer_cache is a radix tree implementation for the kernel, basically it could be being used by anything in the kernel.

Googling is suggestive that this is something normally to do with storage, and being that other slabs in your system are also relatively heavily used for storage, you can see if this works:-

echo 2 >/proc/sys/vm/drop_caches

That will clear out your page cache (which frankly isn't very much), but that might if your lucky clear out the idr_layer_cache slab entries. Storage and page cache nearly always dominates slabs so this is nothing more than a guess to see if that helps really.

Then again -- it might not. There is either some internally compiled kernel module, or externally used kernel module using up that memory.

If that does work, you can try updating your kernel as the bug may have been resolved at some point in the past (and your kernels future).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.