I'm trying to set up a basic high-availability solution with HAproxy and mysql. Currently it looks like this

In case of host1 failing, the host2 is supposed to get it's IP and traffic (using feature of my VPS provider called fail-over IP. I don't think it's possible to get CARP working there).

I want to run the monitoring software on the nodes. Can this work? I'm afraid that in case of the problem with the link between nodes (split-brain situation) each instance of the monitoring software would assume that the other node is dead and try to grab the floating IP for itself.

That's not terrible, but it seems to get worse. With no link between servers and both hosts thinking, that they are the only one alive the database will quickly become inconsistent.

Is there any way out of this situation that doesn't involve third server? Should I really care about a split-brain situation if:

  • Two servers are located in two datacenters of the same provider (just 3 hops away from each other).
  • It seems (traceroute) that the machines used to route the traffic between host1 and host2 are the same used for routing the traffic from the internet. So maybe a split-brain situation with actual writes (since the internet is the only source of writes) is impossible?
  • 1
    Check how your host is implementing IP fail-over. If they can already guarantee that one and only one server will have the IP at a time, then you don't need to worry about split-brain scenarios. If you are doing this for yourself, you need a extra vote. This can come from another cluster node, a lightweight "witness" node, or a quorum disk. Depending on how the VPS environment is set up, a shared disk might not be practical. – Terence Johnson Oct 11 '14 at 1:33

This is a reasonable scenario, and one that STONITH (Shoot The Other Node In The Head) is designed specifically to help with.

If you're going to do this, and you want to be very sure that split-brain can never arise, you need to use some out-of-band control, usually a remotely-controllable power bar, to give each node the power to ensure that the other node really definitely is down (by pulling the plug on it) before taking over service.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.