DRBD / Heartbeat on Virtual Machines

Question

Does anyone have experience configuring drbd with heartbeat between 2 virtual linux machines (VMWare Infrastructure)?

The problem I am running into is that heartbeat likes multiple data paths to see its peer node. For instance, it likes to have a network connection to the peer, maybe one to it's gateway, and a serial cable to its peer - improving the likelihood that when it detects a peer outage, it's actually down, and not due to network congestion or something.

On a virtual machine however, the serial port and ethernet port (and all other ports) are virtual - so really, there's only one data path (correct?)

(I know that VMWare supports physical serial cables between devices, but our vm's are hosted remotely, and physical cables would prevent host migrations, which is not acceptable.)

In our case we are seeing timeouts between the heartbeat peers, even though they are running on the same host machine.

How can I configure drbd / heartbeat to be more robust when running on virtual machines

Answer 1

Have you looked whether the VMs complain about dropped interrupts or similar things - maybe the host hardware is just overloaded or not enough ressources are allocated to your VMs?

If it's a flaky or overloaded network, the right thing to do would of course be fixing that; but if your hosting provider is not keen on that, can you use multiple physical paths by attaching multiple bridged networks to different host devices (hopefully on different switches)?

Just using redundant network paths via 802.3ad couldn't hurt in that case, either.

A commenter on another question mentioned split-brain - that's one thing you want to avoid at all cost: Normally a STONITH script would e.g. turn off a networked PDU strip on the other host so that the other host is down for sure; in a VM you might try a script that switches the other VM off via the VMware API.

Finally - maybe DRBD is just not right for your scenario. If you have a SAN, you may want to open the same device on the fabric on both VMs as a raw disk and then run OCFS2 or a similar cluster FS on it. Friends have seen OCFS2 run rock-solid on up to four nodes simultaneously, which would free you up to do multi-node clusters with heartbeat2 instead of being locked in with two-node fail-over like on heartbeat 1 by drbd.

Caveat emptor: heartbeat 2 uses XML config files. Not everyone (e.g., me) likes that.

Answer 2

Not only does DRBD not use a serial cable, it can't! I have no idea what you're talking about!

On top of that, it does not do squat about multiple data paths, it just communicates to the other node through a plain old TCP connection. The kernel's routing, the switches, routers and firewalls deal with this, DRBD has got nothing to do with it.

Answer 3

The idea to have multiple data paths is not new. It is a basic concept to avoid split brain situations.

But you face exactly the same problem on physical servers - I do not understand why you link this question to VMs?

Multiple data paths can be established with physically distinct networking hardware - which also makes sense in big environments, when you separate the back-end from the font-end of the servers. This would give you two networks, which you can also access on VMs.

If DRBD and heartbeat comes into play a third phsically distinct network might make sense for high-speed dedicated data replication (also often recommended for iSCSI) - this could be the third hb-network, too.

Now to my own experience: We have these three separated networks - which are present on the VM-servers and are being bridged to the VMs as well - so they can run heartbeat across three separate lines...

Another possibility to avoid network failures would be bonding devices for the same networks. If there is no SPOF in your networking system - also a good solution (which gives you at least one HA-network).