I'm learning about clustering and high availability techniques, and stumbled upon an article about configuring a network with a pair of servers, using DRBD for replication and heartbeat for monitoring and failover. The article states that I should have 2 NICs on each server: both eth0 go to LAN and both eth1 should be connected to each other through a crossover cable, like in this image:
Image description on the article:
To ensure automatic failover, the heartbeat monitors the primary server as follows: 1. The secondary server continuously monitors the connection to the primary server over the cross cable connecting the two servers. If the primary is not accessible, then the secondary assumes the primary status. 2. The primary server continuously monitors connections to a highly available network device such as the router. If the network device is not accessible, then it relinquishes control to the secondary server. The failover is thus automatic in case of the following scenarios: 1. Network failover for the primary 2. Hardware failure such as power supply, CPU, RAM etc.
This raised the following question:
If heartbeat on the secondary/passive server is monitoring the primary through eth1, what happens if eth1 fails on any of the servers?
It seems to me that heartbeat will think the primary is dead and will activate the secondary. Wouldn't this create a "split brain" condition? Because the primary server is still connected to the LAN through eth0 and working, it was just the heartbeat/replication link (eth1) that got broken. So now we would have two active servers at the same time?
I'm still grasping the concept, please excuse me if I'm talking nonsense.