We wanted to switch all our servers from lighttpd to nginx but while the Web server migration went fine, the static files server migration was disastrous.
We ended up with up to 10 seconds of latency for loading pictures, using this config with nginx:
There are 4 sites, with about 120k visitors/day that consume massively streamed videos and picures.
A hint we had was that after rebooting nginx, loading picture was always really fast. The CPU or the memory consumption was always really minimal, so we though about the disk. We tried to tweak the file descriptors limit but it didn't change anything.
We switched back to lighttpd and we are back in business again, with nearly instantaneous picture loading.
We know we did something wrong, but what ? It would be nice to have only to maintain one server set up, and nginx is really easier to set up than lighttpd.
The configuration files you are seeing here are for nginx on a virtual machine dedicated to static files and video streaming (compile with the mp4 plugin).