Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I currently have a medium sized cluster (15 servers), with 3 nodes running nginx load balancing with ip_hash. The protocol being balanced is a very high rate of connections, so I have my max_fails set pretty high to avoid false positive server down messages.

Is there a way to monitor when nginx marks a server as offline, as well as when it marks one back online? The access/error logs are massive, and I can't find any specific error message to grep/parse out to find out when a server is considered 'bad'.

share|improve this question

1 Answer 1

We tried to do this too for our nginx and ended up learning that NGiNX keeps the upstream status on a per worker basis, so unless you have only one worker it is hard/difficult to track using a module.

However, we did notice that if you have upstream data in your log it will point out when nginx hit more than one backend, this log format:

log_format  main  '$status:$request_time:$upstream_response_time:$pipe:$body_bytes_sent $connection $remote_addr $host $remote_user [$time_local] "$request" "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_addr $upstream_cache_status';

Will normally produce lines like:

304:0.001:0.001:.:0 26572217 ...

However if two upstreams are failing it will start showing:

304:20.537:10.001, 10.002, 0.534:.:0 26572217 ...

It will also put all the upstreams in $upstream_addr too, so you can see which ones are failing too.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.