Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I currently use Amazon S3 for much of my static file serving needs but my monthly bill is getting very expensive. I did some rough calculations using the logs and at peak times, my most expensive Amazon bucket is handling about 100 180 Mbps of traffic. Mostly images under 50K.

S3 is hugely helpful when it comes to storage and redundancy but I don't really need to be paying for bandwidth and GET requests if I can help it. I have plenty of inexpensive bandwidth at my own datacenter, so I configured an nginx server as a caching proxy and then primed the cache with the bulk of my files (about 240 GB) so that my disk wouldn't be writing like crazy on an empty cache.

I tried cutting over and my my server choked.

It looks like my disks were the problem - this machine has 4 x 1 TB SATA disks (Barracuda XT) set up in RAID 10. It's the only thing that I had on hand with enough storage space to be used for this. I'm pretty sure nginx was set up properly as I had already been using it as a caching proxy for another, smaller Amazon bucket. Assuming that this is a reasonable amount of traffic for a single machine, maybe an SSD would be worth a try.

If you handle large amounts of static file serving, what hardware do you use?

additional information

Filesystem: ext4, mounted noatime,barrier=0,data=writeback,nobh (I have battery backup on the controller) Nginx: worker_connections = 4096, worker_rlimit_nofile 16384, worker_processes 8, open_file_cache max=100000 inactive=60m

share|improve this question
    
Are you sure the bottleneck is your disk setup or are you running into paging issues from overflowing system memory? –  Cypher Oct 3 '10 at 21:49
    
Nope - no swapping.. and about 40 GB of free memory. –  Casey Oct 3 '10 at 22:20
    
So... my question wasn't "are my disks the problem"? They are the problem (or... they are so much of a problem that they obscure any other problems). I was just wondering what other people are using to handle this sort of traffic. –  Casey Oct 4 '10 at 14:15
    
Have you determined how large a portion of your files are actively used? Is it the same 5% of files being server 95% of the time or is the distribution more even? Warming your block cache could make a big difference assuming the bulk of your requests is over a set of files that is a good bit smaller than the amount of RAM you have. –  EvilRyry Oct 4 '10 at 15:56
    
One quick thought: did you set noatime? –  BMDan Oct 4 '10 at 17:44

5 Answers 5

up vote 2 down vote accepted

I don't think your disk is the issue. First nginx's ncache uses a disk store for cache. So, disk speed is going to be one potential cause of issues depending on how hot/cold your dataset is, however, I see no reason that you couldn't serve 100mb/sec with the hardware you've mentioned - especially if you're using nginx.

First thing I would guess is your # of worker processes was low, your worker_connections were probably way too low, and you probably didn't have your open_file_cache set high enough. However, none of those settings would cause a high IO Wait nor a spike like that. You say that you are serving <50k images and it looks like 1/4 of your set could easily be buffered by the OS. Nginx is surely not configured optimally.

Varnish handles the problem in a slightly different way using RAM rather than disk for its cache.

Much depends on your dataset, but, based on the data you've given, I don't see any reason for disk IO to have spiked like that. Did you check dmesg and the logs to see if one of your drives encountered some IO errors at the time? The only other thing I can think that might have caused that spike was exceeding nginx's filecache which would have caused it to have to go into a FIFO mode opening new files.

Make sure your filesystem is mounted with noatime which should cut a considerable amount of writeops off your workload.

As an example of a machine that regularly handles 800mb/sec:

# uptime
 11:32:27 up 11 days, 16:31,  1 user,  load average: 0.43, 0.85, 0.82

# free
             total       used       free     shared    buffers     cached
Mem:       8180796    7127000    1053796          0       1152    2397336
-/+ buffers/cache:    4728512    3452284
Swap:      8297568     237940    8059628

Quadcore Xeon:
    Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz

$ ./bw.pl xxx.xxx 2010-09-01 2010-09-30
bw: 174042.60gb

average 543mb/sec, peaks at 810mb/sec

=== START OF INFORMATION SECTION === Model Family:     Seagate Barracuda
7200.12 family Device Model:     ST3500418AS Serial Number:    6VM89L1N
Firmware Version: CC38 User Capacity: 
500,107,862,016 bytes

Linux 2.6.36-rc5 (xxxxxx)   10/04/2010  _x86_64_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.33    0.00    2.40    5.94    0.00   87.33

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             109.61     19020.67       337.28 19047438731  337754190

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.09    0.00    3.40   10.26    0.00   78.25

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             138.52     21199.60       490.02     106210       2455

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.74    0.00    3.25    9.01    0.00   84.00

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             125.00     21691.20       139.20     108456        696

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.75    0.00    3.12   14.02    0.00   78.11

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             154.69     19532.14       261.28      97856       1309

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.81    0.00    3.36    9.48    0.00   80.36

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             112.80     17635.20       309.00      88176       1545

MRTG:

http://imgur.com/KYGp6.png

Dataset:

# du -sh ads
211.0G  ads

# ls|wc -l
679075
share|improve this answer
    
Why would you want ncache to cache static files on disc that are already located on disk? The kernel should do a fine job caching on its own. –  EvilRyry Oct 4 '10 at 15:50
    
His question stated he had Nginx at his data center caching an S3 bucket. My data merely shows that Nginx running on decent hardware is able to handle 8x what his requirements are which was the second part of his question. Which part did you find confusing so I can edit my answer? –  karmawhore Oct 4 '10 at 16:24
    
Awesome - thanks for the real world info. I added some additional info about nginx and filesystem configuration to my question. –  Casey Oct 4 '10 at 16:50
    
Handling about 1/3rd of requests now as an experiment, no problems. My read IOPS are at 120-140. –  Casey Oct 4 '10 at 17:58
    
run iostat -x 5 5 are you hitting 100% (or close to it) on any of the disks? worker_rlimit_nofile/worker_connections seem low for the amount of traffic you're pushing, but, I don't see anything that would cause the issue you had. I don't recall how nginx handles LRU cleanup - is it possible all of your content expired at the time your IO skyrocketed? what was your lifetime set to? –  karmawhore Oct 4 '10 at 20:23

Your. Discs. Suck. Point.

  • Try getting a lot more and a lot faster discs. SAS comes nicely here, as doe Velociraptors.

  • That said, the best would be getting... a SSD.

Your discs probably do around 200 IOPS each. With SAS you can get that up to around 450, with Velocidaptors to about 300. A high end SSD can get you... 50.000 (no joke - I really mean 5 0 0 0 0 0 0) IOPS.

Make the math ;) A single SSD, no RAID, would be about 62 times as fast as your Raid 10 ;)

share|improve this answer

We're serving about 600 Mbps off of a server with SSDs on the backend, and nginx+varnish on the front. The actual processor is a little Intel Atom; we've got four of them behind a LB doing 600 Mbits/sec each (using DSR). Perhaps not appropriate for every situation, but it's been perfect for our use case.

share|improve this answer

Does the machine you are using have enough ram to have the working set of files cached in RAM?

Also - have you looked at something like Varnish? Nginx is great for handling tons of connections - but it's not the ultimate in terms of caching and systems performance.

share|improve this answer

Add much more disks. You can trade single disk speed with number of disks (up to a certain point): maybe you can get the same performance with X expensive SAS 15kRPM disks or with (guessing, not meaningful values) X*2 cheap SATA 7k2RPM disks. You have to do your math and see what's better for you - and that also depends on how much you pay rack space and power at your datacenter.

SSD will give you all the IOPS you'll need but they're not cheap for bulk storage (that's why their primary use case is database like workloads).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.