Is failover with only two servers possible?

Question

I'm trying to set up a basic high-availability solution with HAproxy and mysql. Currently it looks like this

In case of host1 failing, the host2 is supposed to get it's IP and traffic (using feature of my VPS provider called fail-over IP. I don't think it's possible to get CARP working there).

I want to run the monitoring software on the nodes. Can this work? I'm afraid that in case of the problem with the link between nodes (split-brain situation) each instance of the monitoring software would assume that the other node is dead and try to grab the floating IP for itself.

That's not terrible, but it seems to get worse. With no link between servers and both hosts thinking, that they are the only one alive the database will quickly become inconsistent.

Is there any way out of this situation that doesn't involve third server? Should I really care about a split-brain situation if:

• Two servers are located in two datacenters of the same provider (just 3 hops away from each other).
• It seems (traceroute) that the machines used to route the traffic between host1 and host2 are the same used for routing the traffic from the internet. So maybe a split-brain situation with actual writes (since the internet is the only source of writes) is impossible?

Answer 1

This is a reasonable scenario, and one that STONITH (Shoot The Other Node In The Head) is designed specifically to help with.

If you're going to do this, and you want to be very sure that split-brain can never arise, you need to use some out-of-band control, usually a remotely-controllable power bar, to give each node the power to ensure that the other node really definitely is down (by pulling the plug on it) before taking over service.