This page (http://www.remsys.com/nginx-on-1gbps) claims that it can max out a 1gbps line using JBOD setup and no raid. Currently I'm on a 1gbps dedicated port and I'm on raid 10 (4x2TB - the disks that comes with 100tb.com servers).

Currently I'm only peaking at 500-550mbps. wa% would show something around 20% everytime.

I'm looking into maxing out this 1gbps port because I have an unmetered service from them. Do you guys think that the page that I referenced would be better than my current raid setup? Do you guys have any other suggestions on how to max out the performance of this server? TYI.

  • Just because it's unmetered doesn't mean there aren't other servers on their main pipe... – ceejayoz Mar 28 '12 at 16:49
  • It is unmetered and dedicated. It's 100TB, softlayer network. 100TB is a known host. – AnsellC Mar 28 '12 at 16:56
  • 1
    That's what i was going to write, i have yet to see 1Gbps unmetered, real unshared for under 3.000 US$/month – Niko S P Mar 28 '12 at 16:59
  • @AnsellCruz That's irrelevant. Even reputable hosts have a good chance of overselling bandwidth, because customers tend not to sit pegged at the max. A host with 20 servers might have 10 Gbps of bandwidth in their network, but be able to fairly promise 1Gbps per server the majority of the time. – ceejayoz Mar 28 '12 at 17:05
  • Alright thanks for the comments. Then lets assume that its a 1gbps dedicated pipe. The question is disk related, regardless of the line speed. – AnsellC Mar 28 '12 at 17:07

A lone 2 TB disk can fill a gigabit pipe these days. That requires 125 MByte/sec, which is right about what you should expect for a 7200 RPM 2 TB disk.

Now, that assumes you're sending just one big file down the link, and there's no fragmentation in the file. With RAID-10, you at least double the read data rate of a single disk, and if you're using a decent RAID controller, quadruple it. (Write data rate is only doubled with 4-drive RAID-10.) This gives you enough excess disk I/O capacity to deal with a bit of fragmentation or a small amount of concurrent access.

If you benchmark your disk array, you will probably find that read performance drops by an order of magnitude or more for random reads. This is what happens when there is fragmentation, or when you start sending too many simultaneous files. The disk drive heads spend more time traveling between sectors on the hard drive than they spend in place reading data.

The article suggests breaking the data up and giving each disk its own nginx instance, but that only works if you are able to predict a good way of evenly dividing the workload across disks and nginx instances. If your clients demand downloads unpredictably, you could end up concentrating all reads on a single disk, giving worse performance than the RAID configuration.

If you absolutely had to have better performance without changing the hardware configuration and benchmarked series read performance at only 2x that of a single disk, you could shift to RAID-0. You'd have greater risk of disk failure this way, of course.

The article you link to claims nginx has a really bad I/O performance problem with file reads. Perhaps it has since been fixed. If not, you can alleviate it with multiple nginx instances. You don't have to tie them to individual disks as suggested in the article. You could just set up a proxy to distribute requests round-robin fashion. You'd want to benchmark this carefully to tune the proper number of nginx instances for your particular workload. There is no single "right" answer for all situations.

You could also be running into the problems associated with TCP slow start or one of the inefficiencies in HTTP fixed by SPDY. There's not a whole lot you can practically do about these issues today. The solutions require widespread deployment of experimental technology, which due to network effects means you're looking at a rollout time of years, maybe even decades.

If the problem is simply due to trying to serve too many simultaneous users or too few large files, thereby pushing your disks toward the random file access end of the scale and away from the series I/O end, there's nothing you can do short of adding more spindles. That could mean going with a 6+ disk RAID-10 configuration, multiple servers, or some combination.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.