2

I have been reviewing XtraDB clustering and produced a P.o.C. environment on Openstack using 4 instances, which has fallen over during my resilience testing.

Per the pxc documentation: http://www.percona.com/doc/percona-xtradb-cluster/howtos/virt_sandbox.html which covers a 3 node install I opted for a 4th.

  1. Initial setup complete data loading tests passed, with all nodes being updates synchronously using a 1.6GB test sql file to load a database.
  2. Failiure and restore of nodes commenced, this test entailed stoping the mysql service on a node, creating and subsequently dropping a database to test surviving node replication, and starting of the downed node to resync.
    1. This worked fine for nodes 4,3,2.
    2. Node1 which per the pxc documents is essentially a controller, would not rejoin the cluster.

So my questions are as follows:

  1. How to return a controller node to service if surviving nodes have since had data written to them
  2. Using 4 nodes as a reference, is there a way to remove this single point of failure in node1? (if a surviving node restarts with the controller (node1) down/out of sync, that node will also fail).

3 Answers 3

6

Based on your symptoms on node one, you are using

wsrep_cluster_address=gcomm:// 

in your configuration file, which means the node will start a new cluster. You can confirm this with wsrep_cluster_size variable being 1 on node1, and 3 on the others. If you want to join node1 to your already existing cluster, you should specify

wsrep_cluster_address=gcomm://(ip of a running node here)

In this case, node1 will rejoin the cluster.

Some additional thoughts:

  • Because of the quorum mechanism in PXC (Percona Xtradb Cluster), it's not recommended to run it on 4 nodes. It's recommended to use an odd number of nodes, so in case of a network split, one part of the split cluster will be able have majority.

  • Instead of wsrep_cluster_address you can use wsrep_urls in the [mysqld_safe] section.

Disclaimer: I work for Percona.

3
  • As per 4. in my answer to my question, I confirmed 1 -> 2 -> 3 -> 4 -> 1 works, until you stop all nodes simultaneously; which makes sense in this kind of failiure I can't say you'd want automated recovery, without manual verification. w.r.t the quorum mechanics on PXC, could you please a link some documentation to that effect? also I take it Baron / Ewen pointed you to this? :)
    – Oneiroi
    Jul 5, 2012 at 21:38
  • 1
    Instead of wsrep_cluster_address you can use wsrep_urls in the [mysqld_safe] section. <-- is there an important difference between these settings?
    – user144972
    Nov 10, 2012 at 19:35
  • @grpbbelaar yes the setting for wsrep_urls lists nodes in sequence to try to connect to; timeout on one and it will jump to the next. To clarify, the cluster replicates from what I can tell form all nodes, the initial connection is just to retrive the cluster information (node lists and cluster position) to bring the Joining node in sync :) HOWEVER wsrep_urls is deprecated, you should instead use: wsrep_cluster_address=gcomm://server_one,server_two,server_three iirc
    – Oneiroi
    Dec 6, 2012 at 15:22
1

Further research into this issue seems this is a viable method (leaving this answer unaccepted for the moment, incase someone replies with a better setup):

  1. Circular setup
    1. per pxc documentation have all nodes sync from node 1
    2. stop node2 repoint to node3, start node 2
    3. stop node3 repoint to node4, start node 3
    4. stop node1 repoint to node2, start node 1

This setup appears to tollerate the loss of any node via disconnection at least, and on restoration of the node syncs without issue.

-3

If Mysql wont start and the reason is a corrupt DB table.

replicate what the server is doing and grab a good copy from a stopped server for client dbs.

it tars the files from $MYSQLHOME that are db via a nc.

we used scp to move the good files in place and then kicked off the sync again by starting mysql of the bad server.

1
  • Not sure how this help's the original poster.
    – slm
    Apr 6, 2013 at 11:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .