2

We are using nginx as a load balancer/failover for a pair of upstream servers. We started 11/1.

This chart shows the picture. Where there is no dot, there were no 502s on that day:

enter image description here

In the first few days, the logs show a small number of 502 response codes, likely due to tweaking or other activity as we stabilized our nginx configuration. Then we ran for 12 days with no 502s (except one blip 11/13 - again maybe a tweak).

On 11/20, we moved SSL termination from the upstream servers to the edge. From then on we see 502s daily, and the number seems to be growing (as a percentage of all requests)

Yesterday, for the first time since 11/1, we started receiving client complaints.

Though a low percentage (never reaching 1%) of all traffic (~½ million requests per day), they usually bunch up and span ~10-15 seconds. During this time, many users experience degraded functionality or loss of access.

nginx.conf

worker_processes  auto;

events {
    worker_connections  1024;
    use epoll;
    multi_accept on;
}

http {
  include             mime.types;
  default_type        application/octet-stream;
  sendfile            on;
  keepalive_timeout   70;
  keepalive_requests  100000;
  tcp_nopush          on;
  tcp_nodelay         on;

  open_file_cache max=1000 inactive=20s;
  open_file_cache_valid 30s;
  open_file_cache_min_uses 5;
  open_file_cache_errors off;

  gzip on;
  gzip_min_length 1000;
  gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/xml font/eot font/opentype font/otf image/svg+xml image/vnd.microsoft.icon;
  gzip_disable "MSIE [1-6]\.";

  log_format main '$time_iso8601\t$status\t$remote_addr\t$upstream_addr\t$upstream_status\t$scheme\t$request\t$request_time\t$upstream_response_time\t$body_bytes_sent';
  access_log   /var/log/nginx/access.log  main;
  error_log   /var/log/nginx/error.log  error;
  # error_log   /var/log/nginx/error_debug.log

  upstream example {
    server 192.168.1.40:80;
    server 192.168.1.41:80;
  }

  server {
    listen              80;
    listen              443 default ssl;
    server_name         example.com;

#    ssl on;
    ssl_certificate         ssl/example.com.crt;
    ssl_certificate_key     ssl/example.com.key;
    ssl_trusted_certificate ssl/example.com.pem;

    location / {
      proxy_read_timeout      180;
      proxy_pass              http://example;
      proxy_next_upstream     error timeout invalid_header http_500 http_502 http_503 http_504;

      proxy_set_header        Host            $host;
      proxy_set_header        X-Real-IP       $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}  

When a 502 hits, the access log provides these values:

$status: 502
$upstream_addr: 192.168.1.40, example
$upstream_status: 500, 502

or

$status: 502
$upstream_addr: example
$upstream_status: 502

or variations like those.

The error log says:

[error] 21293#21293: *2441745 no live upstreams while connecting to upstream

Installation details:

  • Ubuntu server 16.04.3 LTS
  • nginx version: nginx/1.12.2
  • 2 CPU cores @ 3.00GHz
  • 8 GB RAM
  • 2x10 Gbe NIC
  • 500 GB HDD

My questions:

  • How did moving the cert to the edge increase the instances of 502 errors and how do we fix it?
  • Why is the rate increasing? Actual load is fairly flat. Is this a leak of some kind?

EDIT TO ADD:

  • adding the keepalive (thanks @Owen Garret) did not eliminate the 502s. We will check tonight if it perhaps reduced, then we can adjust the keepalive value accordingly
  • in the meantime, we reverted to terminating SSL on the web server (passthrough). This, so far, has eliminated the 502s.
  • The errors would suggest network issues. More information about the edge configuration would be needed. – Simon Greenwood Nov 29 '17 at 7:34
  • Error 502 inform you about issues with backend connectivity. For example nginx couldn't connect to backend due network problems or backend down. – Alexander Tolkachev Nov 29 '17 at 7:42
3

NGINX is generating the 502 error because it cannot make an http connection when needed to the upstreams (your 'proxy_pass http://example;' config).

The first place to check would be your upstream servers. Check the server error logs and the syslog to look for indications about why they may be failing.

The problem became greater when you changed from proxying the SSL connection using TCP (stream) load balancing, to terminating SSL and making http connections to the upstream? If so, then one effect of this change is that the upstreams may handle more frequent TCP connects:

  • When proxying the SSL connection using TCP (stream) load balancing, all of the requests in the connection are sent to the upstream in the same proxied connection.
  • When terminating the connection and then making new requests to the upstream, NGINX will by default create a new TCP connection for each request.

You can encourage NGINX to hold TCP connections open and reuse them for future requests, by configuring NGINX to use keepalive connections to the upstreams as described. This change may reduce the number of 502 errors.

Add the following to your location block, alongside the proxy_pass directive:

    proxy_http_version 1.1;
    proxy_set_header Connection "";

Add the following to your upstream group configuration:

    keepalive 20;

See here for more details: https://www.nginx.com/blog/load-balancing-with-nginx-plus-part2/#keepalive

  • Thanks! This seems to have fixed it - at least there have been no 502s to date where yesterday by this time there were several. – biscuit314 Nov 29 '17 at 16:37
  • Dang... spoke too soon. There are still 502s coming (the tail|grep was checking only for 500, so we missed the ones that have been coming in all day). We will analyze to see if the keepalive at least reduced the number, then increase accordingly. If so, will re-mark the answer. – biscuit314 Nov 29 '17 at 17:53
  • Bumped keepalive to 200, which drastically reduced the number of 502s. I'm convinced this is the answer. Will tweak to find the right number. Thank you! – biscuit314 Nov 30 '17 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.