This is something that has plagued me for a very long time and I'm really hoping someone could help me out.
I'll keep it short and simple: I have a file server (2x Dual Core Xeon 2.0, 8GB RAM, 132 SCSI HDD) that hosts several thousand small image files, 4-10kb, and getting up to 1000+ requests per second.
I've tried Apache, Nginx and Lighttpd on it and found that Lighttpd is the most fit for the job.
When the web server is off, a simple HD benchmark shows that it can read at a speed of around 170 megabytes/second. However, when the web server is on and serving about 30 megabytes/second this very same HD benchmark says the HD can read at a speed of only 5 megabytes/second, instead of 140 (170 minus 30) megabytes/second as a dummy like me would expect.
Now, even at 1000 requests/second the CPU is doing fine (load is lower than 1) and there's plenty of free memory, which leads me to believe that the bottleneck is in fact the HD.
So, my question is, why? Why does an HD that can supposedly read at 170 megabytes/second bottlenecks at only 30 megabytes/second when serving that through a web server?
My first guess was that seeking and serving several thousand files at once completely kills the HD's performance, instead of only reading/writing a single file at once like on those benchmark tests.
Is that correct? If so, how can I solve this? RAID? More HDs? SSDs?
Thanks in advance!