2

I have place haproxy in front of a three node ES(elasticsearch) cluster. So far the way i check for each node in haproxy is by using httpcheck. Bellow is a snippet of my config:

backend elastic_nodes
balance roundrobin
option forwardfor
option httpchk
http-check expect status 200
server elastic1 10.88.0.101:9200 check port 9200  fall 3 rise 3
server elastic2 10.88.0.102:9200 check port 9200  fall 3 rise 3
server elastic3 10.88.0.103:9200 check port 9200  fall 3 rise 3

So far this check works fine but if the cluster turns red the response code still is "200" (this is correct since http-wise the node is accessible) which will make haproxy consider the backend server healthy.

On the other side, if i check the status of the cluster and marking a node as down upon receiving health status "Red", this will mark all backend servers as down thus disabling the ES service. My problem on this approach is that in the past indeed my cluster was Red but it was still usable since there was just a single shard missing (a days log). In other words, there are cases where ES Red status is not a big issue and you want to still serve ES requests (instead of marking all backend nodes down with haproxy this blocking ES service).

Is there any other approach to this?

  • if the cluster is red, then all nodes will be red so haproxy will see all nodes as down – Mike Dec 23 '16 at 15:12
  • What exactly are you trying to achieve. Which is to say, under what conditions should a single node be taken our of service, and why? One would think that a RED status shouldn't take any nodes offline, but if the backend either doesn't respond to the check (ES service off, node restarting, etc..), or the check fails (some weird error in ES which wouldn't return status 200), then you'd want that node taken out of service. – GregL Dec 23 '16 at 15:28
  • 1
    @GregL I wish to avoid the situation where the node for some reason cannot server requests but the http check is successful (returns 200). – giomanda Dec 23 '16 at 15:49
  • I can't imagine there being any case/time when that would happen. If ES responds with status 200, then it stands to reason it can respond to requests. Otherwise it likely won't respond at all, or at least not with 200. – GregL Dec 23 '16 at 17:22
4

We use HAproxy to balance between two redundant clusters. During normal operation each receives ~50% of traffic; each is provisioned to take 100% when necessary.

We experienced a fault recently based on a failure case we had not planned for: all client and master nodes stayed up, so our cluster was responsive to REST–but all data nodes were temporarily offline, all indices appeared red and empty, and queries against them returned 0 results. But with a 200, following REST convention.

Our simple HAproxy health check failed us in this case; it merely checked for 200s.

I am now investigating use of http-check expect ! string red with a URI that targets the index of interest directly. I haven't used the more advanced http-check features before.

A more expensive check, but, should correctly take the client nodes for a lobotomized cluster out of the pool.

UPDATE (2): I have switched us over to using

option httpchk get /_cat/indices/<index of interest>
http-check expect rstring \b(green|yellow)\b

and it indeed seems like a better test.

(Second revision: using explicit check for green or yellow instead of just not-red, belatedly thought about index entirely missing from _cat fiter..._

  • ...just don't name your index anything with the string "red" in lolz – aaron_in_sf Feb 13 '17 at 23:09
  • 1
    UPDATE: I got worried about the case of index deleted, so switched to the positive check for green or yellow: http-check expect rstring \b(green|yellow)\b Working so far! – aaron_in_sf Feb 17 '17 at 0:40

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.