Note that inode_cache & ext3_inode_cache slabs are very small compared to dentry_cache.
What happens is that slowly and steadily the within a week dentry_cache grows from 1M to ~5-6G
Then I need to run
echo 2 > /proc/sys/vm/drop_caches && echo 0 > /proc/sys/vm/drop_caches
This started to happening one day on all servers hosting some web code - the developers are saying that they have not changed anything related to filesystem access pattern around the time then the problem started.
The system is centos5 with 2.6.18 kernel so I don't have any instrumentation features available th newer kernels. Any I idea how I can debug the problem? maybe with systemtap? This is a ec2 instance - so not even sure that systemtap will work there.