I'm building an application that requires high-availability. We currently run: 4 webservers, 2 varnish servers (both with Pound for SSL proxy), 3 Cassandra nodes.
The varnish servers are used as entry points for the system. We want to configure everything that Varnish A always works, until failure, then Varnish B should receive the requests and distribute them to the webservers.
As Varnish A and Varnish B are not in the same DC and some other demands of our company it wasn't possible to use some Heartbeat like system to use a virual IP.
We decided to solve it with the CNAME record of our website.
We now have something like this:
hac.domain.com CNAME hac1.domain.com (TTL = 60s)
hac1.domain.com A 18.104.22.168 (TTL = 86400s)
hac2.domain.com A 22.214.171.124 (TTL = 86400s)
Our idea was to swap the hac.domain.com CNAME record as we detect a failure. This gives us a maximum downtime of roughly 2 minutes (if no DNS caches are still redirecting to the wrong server). Acceptable for our use case.
The main question: How to detect the failure in a redundant way and prevent false-positives?