Optimization advices to server a lot of small files

Question

I have nginx+php-fpm+apc setup. No DB. In data folder i have a simple php script that when called with filename parameter checks if the file exists, if it does then it gives it away, if not then it downloads it in the same folder and then gives it away. The speed is very important, now it takes 70ms to output the file in browser.

The files are images with size 1-2kb. With this rate in some months there will be some thousands of images in that folder. And this script will be called tens of times per second(maybe more). So im afraid that the server will start to struggle.

Can you advice me :

-Nginx tweaks

-System tweaks

-Should i run some kind of HDD maintenance now and then?

What else would you recommend in this situation?

RAM: 386MB (upgradable) CPU: Xeon E5506 @ 2.13GHz CentOS 6

Update #1 An interesting idea: Don't have PHP check if a file exists -- have nginx try to serve the file directly, then fall back to PHP as a 404 handler. If you leave nginx to do its thing, it should easily serve ~1000 2kb images/sec with no tweaking.

Answer 1

It's hard to be specific since that would require much more info. If the images are not dynamically generated on each request it would make sense not to pass them through a script each time, i.e. make them static. If the images don't change often I would try to cache most of them into memory. You can avoid a lot of disk I/O since many small images can be put in memory due to them being so small.

We have very good experiences with using Varnish for this. It's a so called reverse proxy-cache, which means it will receive incoming requests from the clients, check its cache, then serve the requested object directly from cache or retrieve it from the backend nginx server first. Varnish can easily handle many thousands of requests per second with very low CPU and I/O usage. Response times are very fast. We're getting response times of 20-30ms, and half of that is what it takes the electrons to travel there and back.

Put Varnish in front of nginx and make sure every unique image has a unique URL. Put a few gigabytes of RAM in your server and configure Varnish to use most of it for its cache. If your images are 1-2kB, 1GB or RAM can cache about 500-1000 images. Varnish will work well with its default ruleset, however it's very tunable which gives it an edge over most other proxy-caches.

Make sure your app doesn't set cookies on the domain, or adjust Varnish to throw away cookies for the URL's that point to the images, since it will take the side of caution and not cache the request if it contains cookies. Also, have your app return expire headers (See 'Cache-Control: max-age') that are far enough in the future to make caching useful. It's easy to set a max-age of weeks, and if an object needs to be expired sooner your app can purge the object from Varnish' cache, or just provide the new image on a new URL.

Answer 2

Don't have PHP check if a file exists -- have nginx try to serve the file directly, then fall back to PHP as a 404 handler. If you leave nginx to do its thing, it should easily serve ~1000 2kb images/sec with no tweaking.

Answer 3

Honestly, if you're relying on the underlying OS to tell you if a file exists... you're going to have problems when you reach a high number of files. The more files you have in a directory... the longer such lookups take. Different file-system formats can do a better job of improving on this... but it's still a time-costly endeavor.

When checking if a file exists... the file system will check from top-to-bottom each file until it finds the one you're looking for. with ext2 (for instance) If you're looking for zzzz... you must go through every file from a -> zzzz before you find it..

You'd be better off doing a long list of tweaks to work around that pile of headaches. i.e. caching search results... limit the number of files in directories... parse files into multiple directories... etc...