I have a pair of CentOS Linux servers in each datacenter. They have failover within each datacenter, managed by heartbeat and DRBD (I know these are outdated tools, but they are stable, so there's no desire to change them).
They also have the capability for inter-datacenter switch as well, to make the east datacenter active while west becomes passive. But this is a manual engineering process, and that's okay.
The west datacenter is currently the active one, the east datacenter is passive.
serverA.west <-> serverB.west <-----------> serverA.east <-> serverB.east ACTIVE DATA CENTER PASSIVE DATA CENTER
Servers can run mysqld and a Java application.
The Java application on this server should run only on the Primary host in the active datacenter (i.e. serverA.west). If another instance of the Java application starts on the Secondary host (serverB.west), or on either host in the passive datacenter, there's a risk of split-brain problems.
Today serverA.east rebooted, which caused heartbeat to flip over to serverB.east. Heartbeat then dutifully started the Java app on serverB.east, which we don't want to happen.
Heartbeat also started mysqld on serverB.east, which is correct, because MySQL replication should keep going, replicating the changes from the west datacenter continuously so the east DC is ready to take over when needed.
/etc/ha.d/haresources names the /etc/init.d scripts for mysqld and the Java application as the resources to start.
We want to allow heartbeat to manage the A/B pair in the passive datacenter. It should start mysqld on a failover, but not the Java app. But if the east datacenter is the active one, then heartbeat should start the Java app during a heartbeat-automated failover.
What's a good way to implement this?
What I am hoping for is something that takes one step to configure as we switch the active datacenter from west to east. Ideally, it should be mistake-proof, i.e. it should be guaranteed that exactly one of the datacenters is configured as the active one.