Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 2 down vote favorite

Note that inode_cache & ext3_inode_cache slabs are very small compared to dentry_cache. What happens is that slowly and steadily the within a week dentry_cache grows from 1M to ~5-6G Then I need to run echo 2 > /proc/sys/vm/drop_caches && echo 0 > /proc/sys/vm/drop_caches This started to happening one day on all servers hosting some web code - the developers are saying that they have not changed anything related to filesystem access pattern around the time then the problem started.

The system is centos5 with 2.6.18 kernel so I don't have any instrumentation features available th newer kernels. Any I idea how I can debug the problem? maybe with systemtap? This is a ec2 instance - so not even sure that systemtap will work there.

Thanks Alex

share|improve this question
can you post your /etc/fstab ? is there a non default setting you use ? – silviud Apr 28 '11 at 1:22
Here it is - but fstab was not changed before the problem started to appear [root@www1a ~]# cat /etc/fstab /dev/sda1 / ext3 defaults,noatime 0 0 none /dev/pts devpts gid=5,mode=620 0 0 none /dev/shm tmpfs defaults 0 0 none /proc proc defaults 0 0 none /sys sysfs defaults 0 0 [root@www1a ~]# – Piavlo Apr 30 '11 at 0:00
It's not clear what the problem is. Why do you care how big the dentry cache is? Does something actually go wrong? Or is this a purely cosmetic problem? – David Schwartz Aug 24 '11 at 23:20

1 Answer

up vote 1 down vote

You have a few options. If I were in this situation I would start tracking the stats in:

# cat /proc/sys/fs/dentry-state 
87338   82056   45      0       0       0

Over time to see how fast it is growing. If the rate is somewhat regular I think you could identify possible culprits in two ways. First looking at the output of lsof might indicate that some process is leaving around deleted file handles. Second, you could strace the main resource using applications and look for an excessive number of fs related calls (like open(), stat(), etc).

I am also curious about @David Schwartz's comment. I haven't seen issues where the dentry cache causes the oom to kill things, but maybe that happens if they are all still referenced and active? If that is the case I'm pretty confident lsof would expose the issue.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.