Is there any simple way of telling Linux to keep a directory of files cached in memory?

I realized that Node.Js and Apache are spending most time on file I/O, making users wait if you have a lot of files (especially for script and template files; the database does an OK job of caching things as needed, if tables are not too large). Of course, you are supposed to use Squid or other ways of speeding things up, but that requires a lot of configuration, and cannot be easily done automatically, and my focus is on "deployability" (Even an interactive installation would require too much from your average user. Of course, all security considerations must be taken care of (including automatic patching of the included httpd's configuration).).

Memory mapping files, on the other hand should be a breeze, since you really just want to tell the system: "Hey, just keep that file in memory until I tell you to drop it or until reboot"; or even some intrusive API that gets its hooks right into relevant POSIX or system calls and does exactly that in user space. Is it that simple? I cannot seem to find some simple code snippet or set of instructions to get that done (not for Linux or Windows).

Help would be much appreciated!

migrated from superuser.com May 3 '14 at 15:52

This question came from our site for computer enthusiasts and power users.

  • You might want to read this presentation of Varnish compared to Squid by the author, it covers some of the items you discuss – Bert May 3 '14 at 17:14
  • Do you have any previous logs indicating that file I/O is a concern to you? You are offering explanations to a solution when you have yet to prevent evidence of a problem. – Matthew Ife May 3 '14 at 17:16
up vote 5 down vote accepted

You can use the vmtouch utility to lock a file or even a directory of files into memory.

vmtouch -dl /var/www/html/important_files

Usage:

vmtouch v0.8.0 - the Virtual Memory Toucher by Doug Hoyte
Portable file system cache diagnostics and control

Usage: vmtouch [OPTIONS] ... FILES OR DIRECTORIES ...

Options:
  -t touch pages into memory
  -e evict pages from memory
  -l lock pages in physical memory with mlock(2)
  -L lock pages in physical memory with mlockall(2)
  -d daemon mode
  -m <size> max file size to touch
  -f follow symbolic links
  -v verbose
  -q quiet

So I can take a directory like this and see how much is in memory:

[root@xt /var/www/html/xt/centos]# vmtouch .
           Files: 146
     Directories: 14
  Resident Pages: 0/5996  0/23M  0%
         Elapsed: 0.059768 seconds

So I can "touch" the directory and bring them into cache:

[root@xt /var/www/html/xt/centos]# vmtouch -t .
           Files: 146
     Directories: 14
   Touched Pages: 5996 (23M)
         Elapsed: 0.64321 seconds

A subsequent check shows:

[root@xt /var/www/html/xt/centos]# vmtouch .
           Files: 146
     Directories: 14
  Resident Pages: 5996/5996  23M/23M  100%
         Elapsed: 0.001846 seconds
  • Exactly what I was looking for. Thanks. I will profile things and see if this helps me getting rid of my bottlenecks. – Domi May 4 '14 at 4:16

Files read from disk are always stored in file cache. The CPU can not access the HD directly. All disk i/o is mapped to virtual memory space, and the CPU accesses files by reading areas of this virtual memory space. If the files aren't already loaded into memory, the kernel retrieves them from the disk and will store the files in memory until memory pressure causes the kernel to empty the file cache.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.