I have place haproxy in front of a three node ES(elasticsearch) cluster. So far the way i check for each node in haproxy is by using httpcheck. Bellow is a snippet of my config:
backend elastic_nodes balance roundrobin option forwardfor option httpchk http-check expect status 200 server elastic1 10.88.0.101:9200 check port 9200 fall 3 rise 3 server elastic2 10.88.0.102:9200 check port 9200 fall 3 rise 3 server elastic3 10.88.0.103:9200 check port 9200 fall 3 rise 3
So far this check works fine but if the cluster turns red the response code still is "200" (this is correct since http-wise the node is accessible) which will make haproxy consider the backend server healthy.
On the other side, if i check the status of the cluster and marking a node as down upon receiving health status "Red", this will mark all backend servers as down thus disabling the ES service. My problem on this approach is that in the past indeed my cluster was Red but it was still usable since there was just a single shard missing (a days log). In other words, there are cases where ES Red status is not a big issue and you want to still serve ES requests (instead of marking all backend nodes down with haproxy this blocking ES service).
Is there any other approach to this?