4

I got 2 servers configured with drbd, corosync with pacemaker. It is all working fine, primary server gets secondary if it fails, and secondary gets primary.

But if the 1st server comes up again after the failover, drbd won't connect back again and I have to do a

drbdadm connect all

on the now secondary.

Here is the log from server1 with all the inputs I made ( https://www.refheap.com/95997 ):

root@server1:~# ifdown eth4


Jan 12 08:11:37 server1 kernel: [237958.935578] ixgbe 0000:04:00.0: removed PHC on eth4
Jan 12 08:11:43 server1 kernel: [237965.430849] cib[2877]: segfault at 0 ip 00007f7d342c8aca sp 00007fffaba39e58 error 4 in libc-2.19.so[7f7d34247000+19f000]
Jan 12 08:11:44 server1 kernel: [237966.427423] crmd[2878]: segfault at 0 ip 00007eff7d291aca sp 00007fffaae51538 error 4 in libc-2.19.so[7eff7d210000+19f000]
Jan 12 08:11:46 server1 kernel: [237967.955014] drbd r0: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Jan 12 08:11:46 server1 kernel: [237967.955124] block drbd1: new current UUID 913B80AB1E998111:B917764AA4AA3175:A9C5E158DC3CB036:A9C4E158DC3CB036
Jan 12 08:11:46 server1 kernel: [237967.955133] drbd r0: asender terminated
Jan 12 08:11:46 server1 kernel: [237967.955137] drbd r0: Terminating drbd_a_r0
Jan 12 08:11:46 server1 kernel: [237967.955278] drbd r0: Connection closed
Jan 12 08:11:46 server1 kernel: [237967.955418] drbd r0: conn( NetworkFailure -> Unconnected )
Jan 12 08:11:46 server1 kernel: [237967.955420] drbd r0: receiver terminated
Jan 12 08:11:46 server1 kernel: [237967.955423] drbd r0: Restarting receiver thread
Jan 12 08:11:46 server1 kernel: [237967.955424] drbd r0: receiver (re)started
Jan 12 08:11:46 server1 kernel: [237967.955442] drbd r0: conn( Unconnected -> WFConnection )
Jan 12 08:11:46 server1 kernel: [237967.955757] drbd r0: conn( WFConnection -> Disconnecting )
Jan 12 08:11:46 server1 kernel: [237967.955809] drbd r0: Connection closed
Jan 12 08:11:46 server1 kernel: [237967.955931] drbd r0: conn( Disconnecting -> StandAlone )
Jan 12 08:11:47 server1 kernel: [237968.955403] drbd r0: receiver terminated
Jan 12 08:11:47 server1 kernel: [237968.955405] drbd r0: Terminating drbd_r_r0

root@server1:~# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
srcversion: 107E17F432EA25ED3AF8929

 1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----
    ns:36 nr:32 dw:136 dr:15337 al:3 bm:8 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0


root@server1:~# crm status
Last updated: Mon Jan 12 08:12:13 2015
Last change: Fri Jan  9 15:37:31 2015 via crmd on server1
Stack: classic openais (with plugin)
Current DC: server1 - partition WITHOUT quorum
Version: 1.1.10-42f2063
2 Nodes configured, 2 expected votes
3 Resources configured


Online: [ server1 ]
OFFLINE: [ server2 ]

 Master/Slave Set: masterdrbd [drbd]
     Masters: [ server1 ]
     Stopped: [ server2 ]
 Resource Group: complete_start
     drbd_mount (ocf::heartbeat:Filesystem):    Started server1





Jan 12 08:12:23 server1 kernel: [238005.311159] ixgbe 0000:04:00.0: registered PHC device on eth4
Jan 12 08:12:23 server1 kernel: [238005.413333] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready
Jan 12 08:12:23 server1 kernel: [238005.477077] ixgbe 0000:04:00.0 eth4: detected SFP+: 6
Jan 12 08:12:24 server1 kernel: [238005.971742] ixgbe 0000:04:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Jan 12 08:12:24 server1 kernel: [238005.971957] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
Jan 12 08:12:27 server1 kernel: [238009.617680] cib[4299]: segfault at 0 ip 00007f8d74805aca sp 00007fffd2ccd9d8 error 4 in libc-2.19.so[7f8d74784000+19f000]
Jan 12 08:12:28 server1 kernel: [238010.612151] crmd[4309]: segfault at 0 ip 00007fb37c610aca sp 00007fff3c4fd5e8 error 4 in libc-2.19.so[7fb37c58f000+19f000]
Jan 12 08:12:33 server1 kernel: [238014.719195] block drbd1: role( Primary -> Secondary )
Jan 12 08:12:33 server1 kernel: [238014.720047] block drbd1: bitmap WRITE of 2 pages took 0 jiffies
Jan 12 08:12:33 server1 kernel: [238014.720058] block drbd1: 8 KB (2 bits) marked out-of-sync by on disk bit-map.



root@server1:~# ifup eth4
root@server1:~# crm status
Last updated: Mon Jan 12 08:12:45 2015
Last change: Fri Jan  9 13:56:26 2015 via crmd on server1
Stack: classic openais (with plugin)
Current DC: server2 - partition with quorum
Version: 1.1.10-42f2063
2 Nodes configured, 2 expected votes
3 Resources configured


Online: [ server1 server2 ]

 Master/Slave Set: masterdrbd [drbd]
     Masters: [ server2 ]
     Slaves: [ server1 ]
 Resource Group: complete_start
     drbd_mount (ocf::heartbeat:Filesystem):    Started server2


root@server1:~# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
srcversion: 107E17F432EA25ED3AF8929

 1: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown   r-----
    ns:36 nr:32 dw:148 dr:15337 al:3 bm:8 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:8



root@server1:~# drbdadm connect all


Jan 12 08:13:14 server1 kernel: [238055.707157] drbd r0: conn( StandAlone -> Unconnected )
Jan 12 08:13:14 server1 kernel: [238055.707190] drbd r0: Starting receiver thread (from drbd_w_r0 [981])
Jan 12 08:13:14 server1 kernel: [238055.707296] drbd r0: receiver (re)started
Jan 12 08:13:14 server1 kernel: [238055.707316] drbd r0: conn( Unconnected -> WFConnection )
Jan 12 08:13:14 server1 kernel: [238056.205606] drbd r0: Handshake successful: Agreed network protocol version 101
Jan 12 08:13:14 server1 kernel: [238056.205613] drbd r0: Agreed to support TRIM on protocol level
Jan 12 08:13:14 server1 kernel: [238056.205656] drbd r0: conn( WFConnection -> WFReportParams )
Jan 12 08:13:14 server1 kernel: [238056.205672] drbd r0: Starting asender thread (from drbd_r_r0 [4701])
Jan 12 08:13:14 server1 kernel: [238056.260475] block drbd1: drbd_sync_handshake:
Jan 12 08:13:14 server1 kernel: [238056.260483] block drbd1: self 913B80AB1E998110:B917764AA4AA3175:A9C5E158DC3CB036:A9C4E158DC3CB036 bits:2 flags:0
Jan 12 08:13:14 server1 kernel: [238056.260487] block drbd1: peer F8EB7C91BB3D8B5D:B917764AA4AA3174:A9C5E158DC3CB036:A9C4E158DC3CB036 bits:3 flags:0
Jan 12 08:13:14 server1 kernel: [238056.260490] block drbd1: uuid_compare()=100 by rule 90
Jan 12 08:13:14 server1 kernel: [238056.260497] block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1
Jan 12 08:13:14 server1 kernel: [238056.262654] block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 exit code 0 (0x0)
Jan 12 08:13:14 server1 kernel: [238056.262680] block drbd1: Split-Brain detected, 1 primaries, automatically solved. Sync from peer node
Jan 12 08:13:14 server1 kernel: [238056.262688] block drbd1: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate )
Jan 12 08:13:14 server1 kernel: [238056.321303] block drbd1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 31(1), total 31; compression: 100.0%
Jan 12 08:13:14 server1 kernel: [238056.374084] block drbd1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 31(1), total 31; compression: 100.0%
Jan 12 08:13:14 server1 kernel: [238056.374100] block drbd1: conn( WFBitMapT -> WFSyncUUID )
Jan 12 08:13:14 server1 kernel: [238056.376960] block drbd1: updated sync uuid B918764AA4AA3174:0000000000000000:A9C5E158DC3CB036:A9C4E158DC3CB036
Jan 12 08:13:14 server1 kernel: [238056.377094] block drbd1: helper command: /sbin/drbdadm before-resync-target minor-1
Jan 12 08:13:14 server1 kernel: [238056.379576] block drbd1: helper command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
Jan 12 08:13:14 server1 kernel: [238056.379603] block drbd1: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
Jan 12 08:13:14 server1 kernel: [238056.379616] block drbd1: Began resync as SyncTarget (will sync 12 KB [3 bits set]).
Jan 12 08:13:14 server1 kernel: [238056.438399] block drbd1: Resync done (total 1 sec; paused 0 sec; 12 K/sec)
Jan 12 08:13:14 server1 kernel: [238056.438410] block drbd1: updated UUIDs F8EB7C91BB3D8B5C:0000000000000000:B918764AA4AA3174:B917764AA4AA3174
Jan 12 08:13:14 server1 kernel: [238056.438418] block drbd1: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
Jan 12 08:13:14 server1 kernel: [238056.438506] block drbd1: helper command: /sbin/drbdadm after-resync-target minor-1
Jan 12 08:13:14 server1 kernel: [238056.440992] block drbd1: helper command: /sbin/drbdadm after-resync-target minor-1 exit code 0 (0x0)


root@server1:~# cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
srcversion: 107E17F432EA25ED3AF8929

 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:12 dw:160 dr:15337 al:3 bm:10 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

and the server2 log with inputs ( https://www.refheap.com/95998 ):

root@server1:~# ifdown eth4



Jan 12 08:03:47 server2 kernel: [234819.924834] ixgbe 0000:04:00.0 eth4: NIC Link is Down
Jan 12 08:03:55 server2 kernel: [234827.977628] drbd r0: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Jan 12 08:03:55 server2 kernel: [234827.977750] drbd r0: asender terminated
Jan 12 08:03:55 server2 kernel: [234827.977754] drbd r0: Terminating drbd_a_r0
Jan 12 08:03:55 server2 kernel: [234827.977831] drbd r0: Connection closed
Jan 12 08:03:55 server2 kernel: [234827.977849] drbd r0: conn( NetworkFailure -> Unconnected )
Jan 12 08:03:55 server2 kernel: [234827.977852] drbd r0: receiver terminated
Jan 12 08:03:55 server2 kernel: [234827.977854] drbd r0: Restarting receiver thread
Jan 12 08:03:55 server2 kernel: [234827.977856] drbd r0: receiver (re)started
Jan 12 08:03:55 server2 kernel: [234827.977869] drbd r0: conn( Unconnected -> WFConnection )
Jan 12 08:03:56 server2 kernel: [234828.077287] block drbd1: role( Secondary -> Primary )
Jan 12 08:03:56 server2 kernel: [234828.077463] block drbd1: new current UUID F8EB7C91BB3D8B5D:B917764AA4AA3174:A9C5E158DC3CB036:A9C4E158DC3CB036
Jan 12 08:03:56 server2 kernel: [234828.359819] EXT4-fs (drbd1): recovery complete
Jan 12 08:03:56 server2 kernel: [234828.359942] EXT4-fs (drbd1): mounted filesystem with ordered data mode. Opts: (null)


root@server2:~#  cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
srcversion: 107E17F432EA25ED3AF8929

 1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
    ns:0 nr:36 dw:4224 dr:6959 al:3 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:12
root@server2:~# crm status
Last updated: Mon Jan 12 08:04:12 2015
Last change: Fri Jan  9 13:56:26 2015 via crmd on server1
Stack: classic openais (with plugin)
Current DC: server2 - partition WITHOUT quorum
Version: 1.1.10-42f2063
2 Nodes configured, 2 expected votes
3 Resources configured


Online: [ server2 ]
OFFLINE: [ server1 ]

 Master/Slave Set: masterdrbd [drbd]
     Masters: [ server2 ]
     Stopped: [ server1 ]
 Resource Group: complete_start
     drbd_mount (ocf::heartbeat:Filesystem):    Started server2

root@server1:~# ifup eth4

Jan 12 08:04:34 server2 kernel: [234866.881710] ixgbe 0000:04:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Jan 12 08:04:43 server2 kernel: [234875.776456] EXT4-fs (drbd1): mounted filesystem with ordered data mode. Opts: (null)


root@server2:~#  cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
srcversion: 107E17F432EA25ED3AF8929

 1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
    ns:0 nr:36 dw:4232 dr:8624 al:3 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:12


root@server1:~# drbdadm connect all


Jan 12 08:05:25 server2 kernel: [234917.115735] drbd r0: Handshake successful: Agreed network protocol version 101
Jan 12 08:05:25 server2 kernel: [234917.115746] drbd r0: Agreed to support TRIM on protocol level
Jan 12 08:05:25 server2 kernel: [234917.115801] drbd r0: conn( WFConnection -> WFReportParams )
Jan 12 08:05:25 server2 kernel: [234917.115807] drbd r0: Starting asender thread (from drbd_r_r0 [2322])
Jan 12 08:05:25 server2 kernel: [234917.170741] block drbd1: drbd_sync_handshake:
Jan 12 08:05:25 server2 kernel: [234917.170749] block drbd1: self F8EB7C91BB3D8B5D:B917764AA4AA3174:A9C5E158DC3CB036:A9C4E158DC3CB036 bits:3 flags:0
Jan 12 08:05:25 server2 kernel: [234917.170753] block drbd1: peer 913B80AB1E998110:B917764AA4AA3175:A9C5E158DC3CB036:A9C4E158DC3CB036 bits:2 flags:0
Jan 12 08:05:25 server2 kernel: [234917.170756] block drbd1: uuid_compare()=100 by rule 90
Jan 12 08:05:25 server2 kernel: [234917.170763] block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1
Jan 12 08:05:25 server2 kernel: [234917.173313] block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 exit code 0 (0x0)
Jan 12 08:05:25 server2 kernel: [234917.173334] block drbd1: Split-Brain detected, 1 primaries, automatically solved. Sync from this node
Jan 12 08:05:25 server2 kernel: [234917.173345] block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
Jan 12 08:05:25 server2 kernel: [234917.231440] block drbd1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 31(1), total 31; compression: 100.0%
Jan 12 08:05:25 server2 kernel: [234917.284423] block drbd1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 31(1), total 31; compression: 100.0%
Jan 12 08:05:25 server2 kernel: [234917.284433] block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1
Jan 12 08:05:25 server2 kernel: [234917.286897] block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0)
Jan 12 08:05:25 server2 kernel: [234917.286926] block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
Jan 12 08:05:25 server2 kernel: [234917.286942] block drbd1: Began resync as SyncSource (will sync 12 KB [3 bits set]).
Jan 12 08:05:25 server2 kernel: [234917.287017] block drbd1: updated sync UUID F8EB7C91BB3D8B5D:B918764AA4AA3174:B917764AA4AA3174:A9C5E158DC3CB036
Jan 12 08:05:25 server2 kernel: [234917.348714] block drbd1: Resync done (total 1 sec; paused 0 sec; 12 K/sec)
Jan 12 08:05:25 server2 kernel: [234917.348722] block drbd1: updated UUIDs F8EB7C91BB3D8B5D:0000000000000000:B918764AA4AA3174:B917764AA4AA3174
Jan 12 08:05:25 server2 kernel: [234917.348732] block drbd1: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )


root@server2:~#  cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
srcversion: 107E17F432EA25ED3AF8929

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:12 nr:36 dw:4232 dr:8636 al:3 bm:7 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

The configuration on the servers are like this:

/etc/drbd.d/r0.res
resource r0 {
        on server1 {
                device /dev/drbd1;
                disk /dev/server1-vg/drbd1;
                address 192.168.1.1:7789;
                meta-disk internal;
        }
        on server2 {
                device /dev/drbd1;
                disk /dev/server2-vg/drbd1;
                address 192.168.1.2:7789;
                meta-disk internal;
        }
#        handlers {
#               split-brain "/usr/lib/drbd/notify-split-brain.sh root";
#       }
        net {
                max-buffers 8000;
                max-epoch-size 8000;
#               sndbuf-size 1024k;
                after-sb-0pri   discard-zero-changes;
                after-sb-1pri   discard-secondary;
                after-sb-2pri   disconnect;
        }
#       disk {
#               resync-rate 4194304;
#       }
#       startup {
#               become-primary-on server1;
#       }
}


/etc/drbd.d/global_common.conf
# DRBD is the result of over a decade of development by LINBIT.
# In case you need professional services for DRBD or have
# feature requests visit http://www.linbit.com

global {
        usage-count no;
        # minor-count dialog-refresh disable-ip-verification
}

common {
#       syncer {
#               rate 1150M;
#       }
        handlers {
                # These are EXAMPLE handlers only.
                # They may have severe implications,
                # like hard resetting the node under certain circumstances.
                # Be careful when chosing your poison.

                # pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                # pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                # local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
                # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
                # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
        }

        startup {
                # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
        }

        options {
                # cpu-mask on-no-data-accessible
        }

        disk {
                # size on-io-error fencing disk-barrier disk-flushes
                # disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
        }

        net {
                # protocol timeout max-epoch-size max-buffers unplug-watermark
                # connect-int ping-int sndbuf-size rcvbuf-size ko-count
                # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
                # after-sb-1pri after-sb-2pri always-asbp rr-conflict
                # ping-timeout data-integrity-alg tcp-cork on-congestion
                # congestion-fill congestion-extents csums-alg verify-alg
                # use-rle
        }
}



crm configure show
node server1
node server2
primitive drbd ocf:linbit:drbd \
        params drbd_resource="r0"
primitive drbd_mount ocf:heartbeat:Filesystem \
        params device="/dev/drbd1" directory="/drbd" fstype="ext4"
group complete_start drbd_mount
ms masterdrbd drbd \
        meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
colocation mountDRBD inf: complete_start masterdrbd:Master
order mouten_danach inf: masterdrbd:promote complete_start:start
property $id="cib-bootstrap-options" \
        dc-version="1.1.10-42f2063" \
        cluster-infrastructure="classic openais (with plugin)" \
        expected-quorum-votes="2" \
        stonith-enabled="false" \
        no-quorum-policy="ignore" \
        default-resource-stickiness="100" \
        last-lrm-refresh="1420808058"

Did I forget something in my configuration? I don't want to do a

drbdadm connect all

on the servers if one fails. This is why I do want high availability. Can you help me please? I don't know what is wrong in my configuration.

1

What distribution are you using?

Make sure you don't have DRBD set to start at boot:

  • CentOS/RHEL 6: chkconfig drbd off
  • CentOS/RHEL 7: systemctl disable drbd
  • Ubuntu/Debian: update-rc.d drbd disable

If it's not set to start at boot, perhaps your network is being torn down before Pacemaker fully stops (I've seen this on Ubuntu clusters).

Check the logs for messages saying the network is coming down before Corosync/Pacemaker finishes stopping resources.

Or, just make sure Pacemaker's init script has the correct INIT INFO:

# grep -A 10 "BEGIN INIT INFO" /etc/init.d/pacemaker 
  ### BEGIN INIT INFO
  # Provides:             pacemaker
  # Required-Start:       $network $remote_fs corosync
  # Should-Start:         $syslog
  # Required-Stop:        $network $remote_fs corosync
  # Default-Start:
  # Default-Stop:
  # Short-Description:    Starts and stops Pacemaker Cluster Manager.
  # Description:          Starts and stops Pacemaker Cluster Manager.
  ### END INIT INFO

Hope that helps find the problem!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.