Using haproxy 1.6.1 (magnificent software!) with the (partial) conf file below. The problem occurs when a backend server is shut down by AWS (spot instance) and replaced. After the new instance is up and running, the stats page still shows a failed health check. tcpdump shows successful outgoing queries to 172.16.0.23 for zos-e-e01.mycompany.com. It appears that haproxy is querying for the ip address, getting the correct response, but not updating the running configuration. Reloading haproxy resolves the issue.
resolvers aws nameserver aws1 172.16.0.23:53 resolve_retries 3 timeout retry 1s hold valid 30s backend zos cookie ZOS insert indirect nocache option httpchk GET /robots.txt HTTP/1.1\r\nHost:\ zos.mycompany.com balance leastconn no log option tcp-smart-connect server zos-e-c01 zos-e-c01.mycompany.com:80 cookie balancer.zos-e-c01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws server zos-e-d01 zos-e-d01.mycompany.com:80 cookie balancer.zos-e-d01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws server zos-e-e01 zos-e-e01.mycompany.com:80 cookie balancer.zos-e-e01 check port 80 inter 2000 rise 2 fall 3 maxconn 60 check resolvers aws