Identifying the Bottleneck on a Static File Server at 1000 requests/sec

Question

This is something that has plagued me for a very long time and I'm really hoping someone could help me out.

I'll keep it short and simple: I have a file server (2x Dual Core Xeon 2.0, 8GB RAM, 132 SCSI HDD) that hosts several thousand small image files, 4-10kb, and getting up to 1000+ requests per second.

I've tried Apache, Nginx and Lighttpd on it and found that Lighttpd is the most fit for the job.

When the web server is off, a simple HD benchmark shows that it can read at a speed of around 170 megabytes/second. However, when the web server is on and serving about 30 megabytes/second this very same HD benchmark says the HD can read at a speed of only 5 megabytes/second, instead of 140 (170 minus 30) megabytes/second as a dummy like me would expect.

Now, even at 1000 requests/second the CPU is doing fine (load is lower than 1) and there's plenty of free memory, which leads me to believe that the bottleneck is in fact the HD.

So, my question is, why? Why does an HD that can supposedly read at 170 megabytes/second bottlenecks at only 30 megabytes/second when serving that through a web server?

My first guess was that seeking and serving several thousand files at once completely kills the HD's performance, instead of only reading/writing a single file at once like on those benchmark tests.

Is that correct? If so, how can I solve this? RAID? More HDs? SSDs?

Thanks in advance!

Answer 1

This does indeed sound like you're saturating your disk's I/O capabilities. A 15K RPM disk can do about 170ish I/O fully random I/O operations a second. When used in a RAID0, RAID1, or RAID10 array this I/O op count is additive based on the number of drives in the array (R5 and R6 introduce another bottleneck so true throughput may be reduced from theoretical). If you have 96 drives in there, your theoretical max is about 16K I/O Ops, sec.

Aside: compare this to even mid-range SSDs these days, which can handle 30K I/O Ops/sec on a single device.

That HD is probably old enough to have a 512b sector size, even though the filesystem probably has a 4kb block size. So you're going to get some sequentiality in your I/O for all of those 4-10kb files. Even so, 1000 concurrent requests a second sounds like it would really saturate a single disk. The fact that your benchmarking during busy times shows a meager 5MB/s tells me that you're saturating the disk.

If your dataset is small enough, a single SSD (or pair in a R1 mirror) will be fast enough to keep up without having to increase your RAM for cache. If that "132 SCSI HD" is actually "132GB SCSI HD", then you're well into the 'somewhat affordable' SSD price-range.

Answer 2

Your problem is indeed most likely seek overhead. There are two major solutions:

• Adding enough RAM to fit your working set into memory is the ideal approach, and quite cheap these days. Even if it looks like you have enough RAM, the problem is not having enough cache RAM, which usually doesn't show as 'used'.
• Failing that, a SSD has much faster seeks than a regular HDD, and may help if you have a working set too large to fit in RAM (ie, too large for your motherboard's RAM limit or would be cheaper than the same amount of ordinary RAM).

RAID10, RAID1 or RAID0 (danger: losing one drive will kill the array) can help split your read accesses across multiple hard drives, improving average access times, but this is only a Nx improvement (where N is the number of drives in use), and so adding RAM should be considered preferable.