nginx loadbalancing - Only log server failures?

Question

I currently have a medium sized cluster (15 servers), with 3 nodes running nginx load balancing with ip_hash. The protocol being balanced is a very high rate of connections, so I have my max_fails set pretty high to avoid false positive server down messages.

Is there a way to monitor when nginx marks a server as offline, as well as when it marks one back online? The access/error logs are massive, and I can't find any specific error message to grep/parse out to find out when a server is considered 'bad'.

Answer 1

We tried to do this too for our nginx and ended up learning that NGiNX keeps the upstream status on a per worker basis, so unless you have only one worker it is hard/difficult to track using a module.

However, we did notice that if you have upstream data in your log it will point out when nginx hit more than one backend, this log format:

log_format  main  '$status:$request_time:$upstream_response_time:$pipe:$body_bytes_sent $connection $remote_addr $host $remote_user [$time_local] "$request" "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_addr $upstream_cache_status';

Will normally produce lines like:

304:0.001:0.001:.:0 26572217 ...

However if two upstreams are failing it will start showing:

304:20.537:10.001, 10.002, 0.534:.:0 26572217 ...

It will also put all the upstreams in $upstream_addr too, so you can see which ones are failing too.