12

I have a small VPS setup with nginx. I want to squeeze as much performance as possible out of it, so I've been experimenting with optimization and load testing.

I'm using Blitz.io to do load testing by GETing a small static text file, and running into an odd issue where the server appears to be sending TCP resets once the number of simultaneous connections reaches roughly 2000. I know this is a very large amount, but from using htop the server still has plenty to spare in CPU time and memory, so I would like to figure out the source of this issue to see if I can push it even further.

I am running Ubuntu 14.04 LTS (64-bit) on a 2GB Linode VPS.

I don't have enough reputation to post this graph directly, so here's a link to the Blitz.io graph:

enter image description here

Here are things I've done to try and figure out the source of the issue:

  • The nginx config value worker_rlimit_nofile is set to 8192
  • have nofile set to 64000 for both hard and soft limits for root and www-data user (what nginx runs as) in /etc/security/limits.conf
  • there are no indications anything is going wrong in /var/log/nginx.d/error.log (typically, if you are running into file descriptor limits, nginx will print error messages saying so)

  • I have ufw setup, but no rate limiting rules. The ufw log indicates nothing is being blocked and I have tried disabling ufw with the same result.

  • There are no indicative errors in /var/log/kern.log
  • There are no indicative errors in /var/log/syslog
  • I have added the following values to /etc/sysctl.conf and loaded them with sysctl -p with no effect:

    net.ipv4.tcp_max_syn_backlog = 1024
    net.core.somaxconn = 1024
    net.core.netdev_max_backlog = 2000
    

Any ideas?

EDIT: I did a new test, ramping to 3000 connections on a very small file (only 3 bytes). Here's the Blitz.io graph:

Blitz.io graph

Again, according to Blitz all of these errors are "TCP Connection reset" errors.

Here's the Linode bandwidth graph. Keep in mind this is a 5 minute average so it's low pass filtered a bit (instantaneous bandwidth is probably much higher), but still, this is nothing:

enter image description here

CPU:

enter image description here

I/O:

enter image description here

Here's htop near the end of the test: htop

I also captured some of the traffic using tcpdump on a different (but similar-looking) test, starting the capture when the errors started coming in: sudo tcpdump -nSi eth0 -w /tmp/loadtest.pcap -s0 port 80

Here's the file if anyone wants to take a look at it (~20MB): https://drive.google.com/file/d/0B1NXWZBKQN6ETmg2SEFOZUsxV28/view?usp=sharing

Here's a bandwidth graph from Wireshark:

enter image description here (Line is all packets, blue bars are TCP errors)

From my interpretation of the capture (and I'm no expert), it looks like the TCP RST flags are coming from the load testing source, not the server. So, assuming that something isn't wrong on the load testing service's side, is it safe to assume that this is the result of some sort of network management or DDOS mitigation between the load testing service and my server?

Thanks!

9
  • Is your provider doing some kind of DDoS mitigation? This may interfere with your test. Apr 1 '15 at 1:43
  • @MichaelHampton I'm fairly certain that Linode doesn't do that.
    – EEAA
    Apr 1 '15 at 1:43
  • Can you post the network graph from the Linode control panel? How much bandwidth is this test actually taking up?
    – EEAA
    Apr 1 '15 at 1:44
  • I did a bit more investigation and updated the original post with lots more info. I also confirmed with Linode that they do not do DDOS mitigation, although this does not necessarily mean that a network provider between the load testing service and Linode isn't doing any. Thanks!
    – Erik Swan
    Apr 1 '15 at 4:47
  • 1
    Is there a reason that you only set net.core.netdev_max_backlog up to 2000? Several examples I've seen have it an order of magnitude higher for gigabit (and 10Gig) connections.
    – Moshe Katz
    Apr 3 '15 at 2:20
2

To set max number of open files ( if that is causing your issue ) you need to add "fs.file-max = 64000" to /etc/sysctl.conf

1

There could be any number of sources of the connection resets. The load tester could be out of available ephemeral ports from which to initiate a connection, a device along the way (such as a firewall doing NAT) might have its NAT pool exhausted and is unable to provide a source port for the connection, is there a load balancer or firewall at your end that might have reached a connection limit? And if doing source NAT on the inbound traffic, that could also experience port exhaustion.

One would really need a pcap file from both ends. What you want to look for is if a connection attempt is sent but never reaches the server but still appears as if it was reset by the server. If that is the case then something along the line had to reset the connection. NAT pool exhaustion is a common source of these kinds of problems.

Also, netstat -st might give you some additional information.

1

Some ideas to try, based on my own recent similar tuning experiences. With references:

You say it's a static text file. Just in case there's any upstream processing going on, apparently domain sockets improve TCP throughput over a TC port based connection:

https://rtcamp.com/tutorials/php/fpm-sysctl-tweaking/ https://engineering.gosquared.com/optimising-nginx-node-js-and-networking-for-heavy-workloads

Regardless of upstream termination:

Enable multi_accept and tcp_nodelay: http://tweaked.io/guide/nginx/

Disable TCP Slow Start: https://stackoverflow.com/questions/17015611/disable-tcp-slow-start http://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum-performance/

Optimize TCP Congestion Window (initcwnd): http://www.nateware.com/linux-network-tuning-for-2013.html

0

Please, look how many ports are in TIME_WAIT state using the command netstat -patunl| grep TIME | wc -l and change net.ipv4.tcp_tw_reuse to 1.

3
  • How would I look at how many ports are in the TIME_WAIT state?
    – Erik Swan
    Apr 1 '15 at 4:52
  • Using netstat or ss. I updated my answer with the complete command!
    – fgbreel
    Apr 1 '15 at 11:30
  • I've rerun the test and watch -n 1 'sudo netstat -patunl | grep TIME | wc -l' returns 0 throughout the entire test. I'm fairy certain the resets are coming as a result of DDOS mitigation by someone between the load tester and my server, based on my analysis of the PCAP file I posted above, but if someone could confirm that would be great!
    – Erik Swan
    Apr 1 '15 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.