Haproxy resolvers not updating dns on failed health check

Question

Using haproxy 1.6.1 (magnificent software!) with the (partial) conf file below. The problem occurs when a backend server is shut down by AWS (spot instance) and replaced. After the new instance is up and running, the stats page still shows a failed health check. tcpdump shows successful outgoing queries to 172.16.0.23 for zos-e-e01.mycompany.com. It appears that haproxy is querying for the ip address, getting the correct response, but not updating the running configuration. Reloading haproxy resolves the issue.

resolvers aws
    nameserver aws1 172.16.0.23:53
    resolve_retries 3
    timeout retry   1s
    hold valid      30s

backend zos
    cookie ZOS insert indirect nocache
    option  httpchk GET /robots.txt HTTP/1.1\r\nHost:\ zos.mycompany.com
    balance leastconn
    no log
    option tcp-smart-connect

    server  zos-e-c01 zos-e-c01.mycompany.com:80 cookie balancer.zos-e-c01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws
    server  zos-e-d01 zos-e-d01.mycompany.com:80 cookie balancer.zos-e-d01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws
    server  zos-e-e01 zos-e-e01.mycompany.com:80 cookie balancer.zos-e-e01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws

Answer 1

There are numerous asynchronous DNS fixes in HAProxy after 1.6.1 but prior to the release of version 1.6.5, which is the current release as of this writing.

I wasn't able to pinpoint a single fix in the changelog that obviously explains exactly the condition you encountered, but if I remember correctly, name resolution at startup is done synchronously, whereas at runtime it's asynchronous, following two different code paths -- so an issue with the much newer async code would not have affected the resolution at startup.

HAProxy 1.6 was still in the process of becoming genuinely stable up through 1.6.5. A lot of new features went into the 1.6 series, and, from reading the mailing list, there were a number of issues encountered in the field that never came up during development and testing. Now that 1.6 is released, no new features are being added -- the only releases should be bugfixes and security fixes -- so it's advisable to upgrade to the latest version within the series.