High Availability/Failover hardware hookup

Question

I'm learning about clustering and high availability techniques, and stumbled upon an article about configuring a network with a pair of servers, using DRBD for replication and heartbeat for monitoring and failover. The article states that I should have 2 NICs on each server: both eth0 go to LAN and both eth1 should be connected to each other through a crossover cable, like in this image:

Image description on the article:

To ensure automatic failover, the heartbeat monitors the primary server as follows: 1. The secondary server continuously monitors the connection to the primary server over the cross cable connecting the two servers. If the primary is not accessible, then the secondary assumes the primary status. 2. The primary server continuously monitors connections to a highly available network device such as the router. If the network device is not accessible, then it relinquishes control to the secondary server. The failover is thus automatic in case of the following scenarios: 1. Network failover for the primary 2. Hardware failure such as power supply, CPU, RAM etc.

This raised the following question:

If heartbeat on the secondary/passive server is monitoring the primary through eth1, what happens if eth1 fails on any of the servers?

It seems to me that heartbeat will think the primary is dead and will activate the secondary. Wouldn't this create a "split brain" condition? Because the primary server is still connected to the LAN through eth0 and working, it was just the heartbeat/replication link (eth1) that got broken. So now we would have two active servers at the same time?

I'm still grasping the concept, please excuse me if I'm talking nonsense.

Answer 1

The article which you're reading is most likely long outdated. Using Heartbeat for resource management (stoping and starting resources upon failover) has been deprecated since about 2008. Pacemaker is the new standard solution for Linux-HA resource management. However, Pacemaker still requires something to handle cluster communication. For the communication layer you may still use Heartbeat, but the most popular solution these days is Corosync.

As for your original question, the short answer is, yes. If you interrupt the network which is carrying the cluster communications it can result in a split-brain, unless, you utilize STONITH, Which you should!. Good articles on why you need STONITH can be found here and here.

STONITH aside, both Heartbeat and Corosync have support for redundant links/networks. Meaning that you may utilize multiple interfaces to ensure that the failure of a single interface does not interfere with cluster communications.

Hope this helps!