I want to put into your consideration the following situation that is currently happening on our active-passive cluster (DRBD, Pacemaker, Corosync, PostgreSQL)
OS: Ubuntu server 14.04 x64 DRBD: 8.4 Pacemaker: 1.1.10 Corosync: 2.3.3 PostgreSQL: 9.3
Here is the problem: When primary goes down, the secondary is elected as primary. The problem comes in at the time of re-introducing the previous primary, it goes directly to a split-brain situation when it should become a secondary instead. Here is the detailed sequence of events and respective logs:
A: Primary, B: Secondary
1- A goes DOWN
2- B becomes PRIMARY
3- A starts again --> SPLIT BRAIN (We presumed that on this case the failover could be done automatically)
LOGS from A:
Jan 28 16:15:11 node1 kernel: [ 538.025422] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0
Jan 28 16:15:11 node1 kernel: [ 538.026185] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
Jan 28 16:15:11 node1 kernel: [ 538.026194] block drbd0: Split-Brain detected but unresolved, dropping connection!
DRBD status:
service drbd status
drbd driver loaded OK; device status:
version: 8.4.3 (api:1/proto:86-101)
srcversion: F97798065516C94BE0F27DC
m:res cs ro ds p mounted fstype
0:pg StandAlone Secondary/Unknown Consistent/DUnknown r-----
AFAIS it seems that DRBD broke the state of this node.
My questions are:
1- Is my assumption correct that on event 3, the returning node can be automatically joint to the cluster?
2- If it can be done, please tell how.