My system run java program only, at some time the memory usage increase very quickly to a very hight percentage:
memory usage of java :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23046 admin 18 0 2115m 427m 13m S 14.7 10.8 1609:58 java
but the system memory usage is :
[root@AY130716093217935dfdZ ~]# free -m total used free shared buffers cached Mem: 3959 3709 249 0 93 122 -/+ buffers/cache: 3493 465 Swap: 0 0 0
then I know the usage exclude java are system kernel's slab :
[root@AY130716093217935dfdZ ~]# cat /proc/meminfo | grep slab -i Slab: 3092688 kB
so, I check the detail usage of slab use slabtop :
Active / Total Objects (% used) : 9509520 / 9545651 (99.6%) Active / Total Slabs (% used) : 771723 / 771725 (100.0%) Active / Total Caches (% used) : 94 / 138 (68.1%) Active / Total Size (% used) : 2751436.50K / 2761816.41K (99.6%) Minimum / Average / Maximum Object : 0.02K / 0.29K / 128.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 3140784 3140358 99% 0.08K 65433 48 261732K selinux_inode_security 3127590 3127260 99% 0.21K 173755 18 695020K dentry_cache 3108264 3108182 99% 0.58K 518044 6 2072176K proc_inode_cache
You can found , the proc_node_cache, entry_cache very large.
so, I suppose there are some program scan the /proc directory ?
but the java program should not do this things.
when I restart the java program, the slab shrink to low.
btw, my system mount an nfs and the java program write log to the file on nfs:
[root@AY130716093217935dfdZ ~]# mount /dev/hda1 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) //188.8.131.52/server23 on /mnt/server23 type cifs (rw,mand) //10.161.143.83/fulllogs on /mnt/newlogs type cifs (rw,mand)
My kernel is :
Linux AY130716093217935dfdZ 2.6.18-308.el5 #1 SMP Tue Feb 21 20:06:06 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
CentOS release 5.8 (Final)
machine was created by XEN. as a XEN guest.
I want to know, what situation will produce such large slab memory usage of proc_inode_cache ?
what maybe happens to my server ?
Thank you !
by view the source code of 2.6.18 kernel. in exit.c:
do_exit -> release_task ->proc_flush_task(p);
the function proc_flush_task:
/** * proc_flush_task - Remove dcache entries for @task from the /proc dcache.
which means when task quit, the proc will be reaped, so, this can explain why slab shrink when I stop the java process.
after now, there is question because the thread of java is 600, which equals to other time, why so much the proc_inode it holds ?