14

I've incorrectly assumed that my internal AB testing means my server can handle 1k concurrency @3k hits per second.

My theory at at the moment is that the network is the bottleneck. The server can't send enough data fast enough.

External testing from blitz.io at 1k concurrency shows my hits/s capping off at 180, with pages taking longer and longer to respond as the server is only able to return 180 per second.

enter image description here

I've served a blank file from nginx and benched it: it scales 1:1 with concurrency.

enter image description here

Now to rule out IO / memcached bottlenecks (nginx normally pulls from memcached), I serve up a static version of the cached page from the filesystem.

enter image description here

The results are very similar to my original test; I'm capped at around 180 RPS.

Splitting the HTML page in half gives me double the RPS, so it's definitely limited by the size of the page.

enter image description here

If I internally ApacheBench from the local server, I get consistent results of around 4k RPS on both the Full Page and the Half Page, at high transfer rates. Transfer rate: 62586.14 [Kbytes/sec] received

If I AB from an external server, I get around 180RPS - same as the blitz.io results.

How do I know it's not intentional throttling?

If I benchmark from multiple external servers, all results become poor which leads me to believe the problem is in MY servers outbound traffic, not a download speed issue with my benchmarking servers / blitz.io.

So I'm back to my conclusion that my server can't send data fast enough.

Am I right? Are there other ways to interpret this data? Is the solution/optimization to set up multiple servers + load balancing that can each serve 180 hits per second?

I'm quite new to server optimization, so I'd appreciate any confirmation interpreting this data.


Outbound traffic

Here's more information about the outbound bandwidth: The network graph shows a maximum output of 16 Mb/s: 16 megabits per second. Doesn't sound like much at all.

Due to a suggestion about throttling, I looked into this and found that linode has a 50mbps cap (which I'm not even close to hitting, apparently). I had it raised to 100mbps.

Since linode caps my traffic, and I'm not even hitting it, does this mean that my server should indeed be capable of outputting up to 100mbps but is limited by some other internal bottleneck? I just don't understand how networks at this large of a scale work; can they literally send data as fast as they can read from the HDD? Is the network pipe that big?

enter image description here


In conclusion

1: Based on the above, I'm thinking I can definitely raise my 180RPS by adding an nginx load balancer on top of a multi nginx server setup at exactly 180RPS per server behind the LB.

2: If linode has a 50/100mbit limit that I'm not hitting at all, there must be something I can do to hit that limit with my single server setup. If I can read / transmit data fast enough locally, and linode even bothers to have a 50mbit/100mbit cap, there must be an internal bottleneck that's not allowing me to hit those caps that I'm not sure how to detect. Correct?

I realize the question is huge and vague now, but I'm not sure how to condense it. Any input is appreciated on any conclusion I've made.

2
  • 1
    To check if it is a bandwidth problem, you could make your html page much larger so that the same bandwidth is reached with much fewer requests. If your page is eg 5MB large, then you should be able to reach the same throughput with only a few requests/second, which should have a lot less overhead and so get you closer to your actual bandwidth limit.
    – brain99
    Aug 29, 2012 at 20:04
  • I just tested a page that's exactly 10x the size. My RPS correlates directly to page size. 10x larger == 18RPS. 1x == 180. I actually think this is suspiciously close to 50mbits. I think there's a chance linode's status monitoring max of 24mbits could be wrong, and I'm actually hitting their cap. I'm asking for an increase again and will report back. Aug 29, 2012 at 20:28

3 Answers 3

5

The issue was me assuming that the linode.com graph peaks were true peaks. It turns out the graph uses 5 minute average data points, thus my peak appeared to be 24mbits when in fact I was hitting the 50mbit cap.

Now that they have raised it to 100mbits, my benchmarks immediately went up to the new outbound traffic limit.

If only I had noticed that earlier! A lot of my reasoning hinged on the idea that I was not hitting an outbound traffic limit due to that graph.

Now, I peak at 370 requests per second, which is right under 100mbps at which point I start getting a "backlog" of requests, and response times start going up.

enter image description here

I can now increase maximum concurrency by shrinking the page; with gzip enabled I get 600RPS.

enter image description here

I do still run into problems when I suddenly peak and the backlog of pending requests (limited by bandwidth) start to accumulate, but that sounds like a different question.

enter image description here

It's been a great lesson in optimization / reading this data / narrowing down the possible problems. Thank you so much for your input!

5

A bit late now that you figured it out... but maybe you should consider reading the ServerFault blog from time to time.

I'm thinking in particular of this post, where they discuss why having a one second polling interval doesn't cut it from time to time, related to a very similar problem to the one you had..

We discovered that we were discarding packets pretty frequently on 1 Gbit/s interfaces at rates of only 10-30 MBit/s which hurts our performance. This is because that 10-30 MBit/s rate is really the number of bits transfered per 5 minutes converted to a one second rate. When we dug in closer with Wireshark and used one millisecond IO graphing, we saw we would frequently burst the 1 Mbit per millisecond rate of the so called 1 Gbit/s interfaces.

Sure made me think. And I just know know I'm busting that one out on the other SAs at my shop the first chance I get, and will look wickedly brilliant and perceptive when we hit this problem.

Who knows, I may even let some of them in the secret. :)

1
  • Good point! Interesting they brought up the 5 minute graph @ 1 second rate too... I'm relatively comfortable with the data because my test of 1k concurrent is already a worst-case peak (i think..). ~600 users loading a page every second == ~2m hits an hour, which we don't get even close to. I just didn't want to bog down in the first few minutes of a spike. Aug 30, 2012 at 16:09
0

It may be limited by the network, but not necessarily simply a question of bandwidth. Latency of your remote test unit will have an effect on the number of connections pending at any given time (waiting 50ms for acknowledgements is a lot different than .5ms locally) as well as on the negotiation and stabilization of window sizes as the connection progresses. You're also likely exposed to some amount of packet loss - either as a function of congestion or as the mechanism of bandwidth limitation on the part of your carrier (or those upstream).

I'd suggest eliminating as much as possible from the equation to draw a sensible baseline. Measure peak bandwidth, latency and packet loss from your server to a few points on the general Internet. As unlikely as it might sound, try searching for "Voip traffic test" or similar. Several providers of VOIP services have apps that can measure these sorts of patterns (bidirectionally) with a fair degree of accuracy. Once you have some valid empirical data as to the actual useful speed of your link then your results may well be validated.

In addition to bandwidth tests it might also be useful to look at a packet capture of the sub-par web traffic to look for excessive numbers of retransmissions as well as measuring the apparent time your server is taking to respond to requests (..if this value is increasing substantially as a function of number of connections, this is a big clue).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .