2

Is there any simple way of telling Linux to keep a directory of files cached in memory?

I realized that Node.Js and Apache are spending most time on file I/O, making users wait if you have a lot of files (especially for script and template files; the database does an OK job of caching things as needed, if tables are not too large). Of course, you are supposed to use Squid or other ways of speeding things up, but that requires a lot of configuration, and cannot be easily done automatically, and my focus is on "deployability" (Even an interactive installation would require too much from your average user. Of course, all security considerations must be taken care of (including automatic patching of the included httpd's configuration).).

Memory mapping files, on the other hand should be a breeze, since you really just want to tell the system: "Hey, just keep that file in memory until I tell you to drop it or until reboot"; or even some intrusive API that gets its hooks right into relevant POSIX or system calls and does exactly that in user space. Is it that simple? I cannot seem to find some simple code snippet or set of instructions to get that done (not for Linux or Windows).

Help would be much appreciated!

  • You might want to read this presentation of Varnish compared to Squid by the author, it covers some of the items you discuss – Bert May 3 '14 at 17:14
  • Do you have any previous logs indicating that file I/O is a concern to you? You are offering explanations to a solution when you have yet to prevent evidence of a problem. – Matthew Ife May 3 '14 at 17:16
5

You can use the vmtouch utility to lock a file or even a directory of files into memory.

vmtouch -dl /var/www/html/important_files

Usage:

vmtouch v0.8.0 - the Virtual Memory Toucher by Doug Hoyte
Portable file system cache diagnostics and control

Usage: vmtouch [OPTIONS] ... FILES OR DIRECTORIES ...

Options:
  -t touch pages into memory
  -e evict pages from memory
  -l lock pages in physical memory with mlock(2)
  -L lock pages in physical memory with mlockall(2)
  -d daemon mode
  -m <size> max file size to touch
  -f follow symbolic links
  -v verbose
  -q quiet

So I can take a directory like this and see how much is in memory:

[root@xt /var/www/html/xt/centos]# vmtouch .
           Files: 146
     Directories: 14
  Resident Pages: 0/5996  0/23M  0%
         Elapsed: 0.059768 seconds

So I can "touch" the directory and bring them into cache:

[root@xt /var/www/html/xt/centos]# vmtouch -t .
           Files: 146
     Directories: 14
   Touched Pages: 5996 (23M)
         Elapsed: 0.64321 seconds

A subsequent check shows:

[root@xt /var/www/html/xt/centos]# vmtouch .
           Files: 146
     Directories: 14
  Resident Pages: 5996/5996  23M/23M  100%
         Elapsed: 0.001846 seconds
| improve this answer | |
  • Exactly what I was looking for. Thanks. I will profile things and see if this helps me getting rid of my bottlenecks. – Domi May 4 '14 at 4:16
2

Files read from disk are always stored in file cache. The CPU can not access the HD directly. All disk i/o is mapped to virtual memory space, and the CPU accesses files by reading areas of this virtual memory space. If the files aren't already loaded into memory, the kernel retrieves them from the disk and will store the files in memory until memory pressure causes the kernel to empty the file cache.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.