3

Using haproxy 1.6.1 (magnificent software!) with the (partial) conf file below. The problem occurs when a backend server is shut down by AWS (spot instance) and replaced. After the new instance is up and running, the stats page still shows a failed health check. tcpdump shows successful outgoing queries to 172.16.0.23 for zos-e-e01.mycompany.com. It appears that haproxy is querying for the ip address, getting the correct response, but not updating the running configuration. Reloading haproxy resolves the issue.

resolvers aws
    nameserver aws1 172.16.0.23:53
    resolve_retries 3
    timeout retry   1s
    hold valid      30s

backend zos
    cookie ZOS insert indirect nocache
    option  httpchk GET /robots.txt HTTP/1.1\r\nHost:\ zos.mycompany.com
    balance leastconn
    no log
    option tcp-smart-connect

    server  zos-e-c01 zos-e-c01.mycompany.com:80 cookie balancer.zos-e-c01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws
    server  zos-e-d01 zos-e-d01.mycompany.com:80 cookie balancer.zos-e-d01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws
    server  zos-e-e01 zos-e-e01.mycompany.com:80 cookie balancer.zos-e-e01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws
  • 1
    HAProxy is up to 1.6.4, and iirc the resolution at startup follows a different code path than the async DNS at runtime... which is new in 1.6 and did have some issues... this may have been fixed since 1.6.1. – Michael - sqlbot May 6 '16 at 2:34
2

There are numerous asynchronous DNS fixes in HAProxy after 1.6.1 but prior to the release of version 1.6.5, which is the current release as of this writing.

I wasn't able to pinpoint a single fix in the changelog that obviously explains exactly the condition you encountered, but if I remember correctly, name resolution at startup is done synchronously, whereas at runtime it's asynchronous, following two different code paths -- so an issue with the much newer async code would not have affected the resolution at startup.

HAProxy 1.6 was still in the process of becoming genuinely stable up through 1.6.5. A lot of new features went into the 1.6 series, and, from reading the mailing list, there were a number of issues encountered in the field that never came up during development and testing. Now that 1.6 is released, no new features are being added -- the only releases should be bugfixes and security fixes -- so it's advisable to upgrade to the latest version within the series.

| improve this answer | |
  • 1
    1.6.4 not only fixed the dns resolvers issue, CPU load dropped from 65-75% to 10-15% on the haproxy instances. Very useful answer, thanks. – mdkerman May 19 '16 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.