We are using nginx as a load balancer/failover for a pair of upstream servers. We started 11/1.
This chart shows the picture. Where there is no dot, there were no 502s on that day:
In the first few days, the logs show a small number of 502 response codes, likely due to tweaking or other activity as we stabilized our nginx configuration. Then we ran for 12 days with no 502s (except one blip 11/13 - again maybe a tweak).
On 11/20, we moved SSL termination from the upstream servers to the edge. From then on we see 502s daily, and the number seems to be growing (as a percentage of all requests)
Yesterday, for the first time since 11/1, we started receiving client complaints.
Though a low percentage (never reaching 1%) of all traffic (~½ million requests per day), they usually bunch up and span ~10-15 seconds. During this time, many users experience degraded functionality or loss of access.
When a 502 hits, the access log provides these values:
$status: 502 $upstream_addr: 192.168.1.40, example $upstream_status: 500, 502
$status: 502 $upstream_addr: example $upstream_status: 502
or variations like those.
The error log says:
[error] 21293#21293: *2441745 no live upstreams while connecting to upstream
- Ubuntu server 16.04.3 LTS
- nginx version: nginx/1.12.2
- 2 CPU cores @ 3.00GHz
- 8 GB RAM
- 2x10 Gbe NIC
- 500 GB HDD
- How did moving the cert to the edge increase the instances of 502 errors and how do we fix it?
- Why is the rate increasing? Actual load is fairly flat. Is this a leak of some kind?
EDIT TO ADD:
- adding the keepalive (thanks @Owen Garret) did not eliminate the 502s. We will check tonight if it perhaps reduced, then we can adjust the keepalive value accordingly
- in the meantime, we reverted to terminating SSL on the web server (passthrough). This, so far, has eliminated the 502s.