Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am dealing with hundred million files in a filesystem (distributed among a lot of subdirectories), and I need to be able to list them very quickly, particularly in order to rsync them efficiently.

On a other hand, I don't really need to have the actual content of the file kept in cache.

I am constantly adding and removing files, but not that frequently (something like ten times per second).

Is there a way I can tell the OS (2.6.18-194.el5) to use the 24GB available RAM more on inode caching than on file caching? I already looked at /proc/etc/vm/vfs_cache_pressure but it doesn't seem to be exactly what I am looking for...

share|improve this question

2 Answers 2

you can use these 2 commands to do the same job.

Updatedb (to update the list of file and folders location in whole drive)

locate / (to list all files in the whole OS, which is lightening fast as it picks them up from the Database)

share|improve this answer
    
Yup, I run updatedb from cron to keep the inode cache 'warm.' It works well, take a look at your slabtop to see statistics. Also, running strace -c updated can give you some insight as far as how much gets updated. –  Marcin Dec 6 '11 at 13:00
    
Thank you, but how does that help me improving rsync performance?edit: Ok, just read Marcin's comment, but regarding the warmness of the cache, how is it different from running a simple find /? –  john.doe Dec 6 '11 at 13:03
    
In some implementations, the updatedb shell script actually runs find / to get the initial file list. On Mac OS X it still works this way. –  Ladadadada Dec 6 '11 at 14:38

How about lowering the value for vfs_cache_pressure? According to Documentation for /proc/sys/vm/* this should do what you want:

vfs_cache_pressure

This percentage value controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

Increasing vfs_cache_pressure significantly beyond 100 may have negative performance impact. Reclaim code needs to take various locks to find freeable directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.