2

since using CentOS 7, we switched from a regular heartbeat setup to pacemker.

Mainly we have IP resources that are active on one node and switch to a second node if a failover happens. Also we execute some scripts in case of a failover. Nothing special.

For resources to always start on the primary node, i use

pcs constraint location Cluster_IP prefers server1=master-server

I also use

pcs resource defaults resource-stickiness=INFINITY

to prevent resources moving back after failover.

This works fine for me, if the master fails (hardware failure for example).

Since its not a problem for me if the failover takes some time, i would like to implement some kind of delay in case of a short split brain.

Before doing anything, the slave should wait ~2 Minutes, before it takes over, in case the master is reachable again in this ~2 Minutes.

I was wondering, what would be the best way to do it?

2 Answers 2

1

I've never set a token timeout in Corosync to anything above 10 seconds, but you could try increasing/setting the token value in your corosync.conf to 120000 (120 seconds in milliseconds). token should be defined in the totem{} section of your corosync.conf; man corosync.conf for more detail.

That should prevent Corosync from declaring a node dead for 120s when the network flakes out.

3
  • @sebkoe: This should work, I quickly tested it on a test cluster with a single ring. I just added a ring to my test cluster, and it took twice as long, so try setting token to half your desired time if you're using rrp_mode: passive. Commented Mar 27, 2018 at 15:14
  • Thanks, it now worked. Maybe i did something wrong in my first try
    – sebkoe
    Commented Mar 28, 2018 at 8:49
  • @sebkoe perhaps you forgot to restart Corosync? Either way, don't forget to mark your answer ;) Commented Mar 28, 2018 at 14:20
1

You can change the monitor interval of the resource with op monitor interval=Ns where N is the number of seconds, and then set the migration-threshold on the resource to 2. Keep in mind with a setting of 120s you could see a total delay of 120-240s depending on when the initial failure occurs in the monitoring interval.

There are other caveats to this, in that the failure counter applied to migration-threshold does not reset on success. To do that you would also need to set failure-timeout or manually intervene.

With op monitor interval=120s, migration-threshold=2, failure-timeout=121s and your resource-stickiness setting, you would need to test to ensure this provides the functionality you expect and how the failure counters behave when the original master recovers. It could require manual intervention but I'm not 100% certain on that

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .