Fast small static file serving on a VPS - avoiding the Disk

Question

I need to serve a large number of very small static files (e.g. 1Million, ~1KB), from a VPS. My current favoured approach is Nginx, but my main concern is hitting slow VPS disks.

Is there an approach I can use which would serve these files from local memory, rather than going to the Disk? I understand that the OS (Debian in this case) will keep 'hot' files in memory, but I'm worried that this will be difficult to monitor or influence. Perhaps there is a way to get Nginx to serve from memory? Or some other approach entirely?

Answer 1

You can put the folder they are in in the memory by mounting it as a tmpfs partition.

For example: mount -t tmpfs -o size=128m tmpfs /mnt/tmp

Answer 2

I hate to be contrary, but don't do this. Assuming you're using Linux, the OS will in any case cache frequently-accessed files in memory that is not more urgently needed by the OS. If you try to second-guess this process, eg by using a ramdisk as Jeroen suggests, you'll either be right or wrong about your files needing to be in memory more than what the kernel might otherwise keep there.

If you're right, then all you've done is what the VM system was going to do for you anyway.

If you're wrong, then some other piece of memory that would have displaced your files will instead be flushed to swap, and the retrieval of those pages from swap will put more of a performance load on your server than the retrieval of your files from the FS would have done.

The VM code in Linux is tightly and intensively tuned. It doesn't always get it right, but it tends to get it right such a large fraction of the time that you need a really good reason to override it.

Answer 3

I understand that the OS (Debian in this case) will keep 'hot' files in memory, but I'm worried that this will be difficult to monitor or influence.

This is true. You can influence this by tuning your virtual memory options;

Read about swappiness here. Set it to 1 or 0 in your case:

echo 0 > /proc/sys/vm/swappiness
# This will set it permanently, to continue after reboot
sysctl -w vm.swappiness=1

You could make sure your cache pressure is 100, or more;

echo 100 > /proc/sys/vm/vfs_cache_pressure
# Set permanently for after restart
sysctl -w vm.vfs_cache_pressure=100

Read about all these options and more here.

Another option is to place all your files on to a ramdisk; a partition that exists in your VPS's memory. See instructions here and here. Obviously for this option though, be aware that the ramdisk is destroyed at reboot. It will be recreated empty, so at start-up, create and mount a ramdisk, then copy files on to it. Don't make that the only copy of them :)

Answer 4

Ramdisk is a good idea. You can also get rad, and try vmtouch to lock a directory into the filesystem cache. It's more elegant since it can be daemonized and eliminates the issue of non-persistent RAM disks.

Lock all files in a directory into physical memory:
vmtouch -dl /var/www/html/directory-of-small-static-files/

Other examples...