2

I currently have a medium sized cluster (15 servers), with 3 nodes running nginx load balancing with ip_hash. The protocol being balanced is a very high rate of connections, so I have my max_fails set pretty high to avoid false positive server down messages.

Is there a way to monitor when nginx marks a server as offline, as well as when it marks one back online? The access/error logs are massive, and I can't find any specific error message to grep/parse out to find out when a server is considered 'bad'.

2

We tried to do this too for our nginx and ended up learning that NGiNX keeps the upstream status on a per worker basis, so unless you have only one worker it is hard/difficult to track using a module.

However, we did notice that if you have upstream data in your log it will point out when nginx hit more than one backend, this log format:

log_format  main  '$status:$request_time:$upstream_response_time:$pipe:$body_bytes_sent $connection $remote_addr $host $remote_user [$time_local] "$request" "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_addr $upstream_cache_status';

Will normally produce lines like:

304:0.001:0.001:.:0 26572217 ...

However if two upstreams are failing it will start showing:

304:20.537:10.001, 10.002, 0.534:.:0 26572217 ...

It will also put all the upstreams in $upstream_addr too, so you can see which ones are failing too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.