Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have an Nginx Reverse Proxy (RP) bound to one IP, that load-balances/fails-over to two IPs that are located roughly 140ms away, by network time. Each of these upstream IPs are on different ISPs, but are bound to interfaces on a PFsense box. The PFsense box uses NAT to translate incoming HTTP connections to an internal Microsoft NLB IP that is then bound to a number of webservers that serve the actual requests. I doubt that the pfsense or iis have anything to do with this, but I am mentioning them just in case you brilliant people have some insight into how it may actually be a part of this.

For baseline traffic of ~10 req/s, this configuration works fine. But our use case deals with traffic spikes that oftentimes go to ~200 req/s. When this happens, all hell breaks loose. The nginx error log begins filling up with tons of the following:

upstream timed out (110: Connection timed out) while connecting to upstream

and occasionally a few of these:

connect() failed (113: No route to host) while connecting to upstream

Pretty soon, nginx collects enough of these that it decides one or both of the upstreams are "down", and then starts emitting 503 or 504 error messages to the clients, but not before slowing down to a completely unusable extent.

During this time, the load average on the box does not exceed 0.15, and cpu usage is close to 5%. Both the reverse proxy and the upstreams have ample available bandwidth.

I have reviewed the answers available at nginx errors: upstream timed out (110: Connection timed out) but they don't reflect my situation. If I take the reverse proxy out of the equation and try to stress the upstream servers directly, the upstream servers handle all requests flawlessly.

Here are the relevant lines in my nginx config:

worker_processes  8;  # I have 8 cores on this box, so this should be right
events {
   worker_connections  768;
}
sendfile        on;
tcp_nopush     on;  # tried both settings, doesn't matter
keepalive_timeout  65;
tcp_nodelay        off;  # tried both settings, doesn't matter
gzip  on;

upstream tcinstances {
   server [ip_removed] weight=1 max_fails=10 fail_timeout=30s; # tried default/different max_fails, same result
   server [ip_removed] weight=1 max_fails=10 fail_timeout=30s;
}

proxy_cache_path  /var/www/cache levels=1:2 keys_zone=tc-cache:4m max_size=128m inactive=600m;
proxy_temp_path /var/www/cache/tmp;

server {
 listen   80;
 server_name  drp;

 proxy_buffering on;  # tried this off, also, doesn't change anything

 proxy_read_timeout 30s;
 proxy_connect_timeout 2s; # I know this looks low, but if I set this to 60s or default, all requests become *extremely slow* immediately, and they eventually time out anyway.  at least this way it fails fast.
 proxy_next_upstream error timeout;

 location / {
      proxy_pass http://tcinstances;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        Host $http_host;
      expires 1s;
      add_header Cache-Control
      public,max-age=604800,post-check=604800,pre-check=1209600;
 }

  location ~* ^.+\.(png|jpg|gif|css|js)$ {
      proxy_pass http://tcinstances;
      proxy_set_header        Host $http_host;
      proxy_cache tc-cache;
      proxy_cache_valid  200 302  60m;
      proxy_cache_valid  404      1m;
      proxy_cache_key "$scheme$host$request_uri";
      access_log off;
     expires 1d;
  }
}

Any help in figuring this out would be absolutely appreciated. Thanks.

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.