I have been reviewing XtraDB clustering and produced a P.o.C. environment on Openstack using 4 instances, which has fallen over during my resilience testing.

Per the pxc documentation: http://www.percona.com/doc/percona-xtradb-cluster/howtos/virt_sandbox.html which covers a 3 node install I opted for a 4th.

  1. Initial setup complete data loading tests passed, with all nodes being updates synchronously using a 1.6GB test sql file to load a database.
  2. Failiure and restore of nodes commenced, this test entailed stoping the mysql service on a node, creating and subsequently dropping a database to test surviving node replication, and starting of the downed node to resync.
    1. This worked fine for nodes 4,3,2.
    2. Node1 which per the pxc documents is essentially a controller, would not rejoin the cluster.

So my questions are as follows:

  1. How to return a controller node to service if surviving nodes have since had data written to them
  2. Using 4 nodes as a reference, is there a way to remove this single point of failure in node1? (if a surviving node restarts with the controller (node1) down/out of sync, that node will also fail).

3 Answers 3


Based on your symptoms on node one, you are using


in your configuration file, which means the node will start a new cluster. You can confirm this with wsrep_cluster_size variable being 1 on node1, and 3 on the others. If you want to join node1 to your already existing cluster, you should specify

wsrep_cluster_address=gcomm://(ip of a running node here)

In this case, node1 will rejoin the cluster.

Some additional thoughts:

  • Because of the quorum mechanism in PXC (Percona Xtradb Cluster), it's not recommended to run it on 4 nodes. It's recommended to use an odd number of nodes, so in case of a network split, one part of the split cluster will be able have majority.

  • Instead of wsrep_cluster_address you can use wsrep_urls in the [mysqld_safe] section.

Disclaimer: I work for Percona.

  • As per 4. in my answer to my question, I confirmed 1 -> 2 -> 3 -> 4 -> 1 works, until you stop all nodes simultaneously; which makes sense in this kind of failiure I can't say you'd want automated recovery, without manual verification. w.r.t the quorum mechanics on PXC, could you please a link some documentation to that effect? also I take it Baron / Ewen pointed you to this? :)
    – Oneiroi
    Jul 5, 2012 at 21:38
  • 1
    Instead of wsrep_cluster_address you can use wsrep_urls in the [mysqld_safe] section. <-- is there an important difference between these settings?
    – user144972
    Nov 10, 2012 at 19:35
  • @grpbbelaar yes the setting for wsrep_urls lists nodes in sequence to try to connect to; timeout on one and it will jump to the next. To clarify, the cluster replicates from what I can tell form all nodes, the initial connection is just to retrive the cluster information (node lists and cluster position) to bring the Joining node in sync :) HOWEVER wsrep_urls is deprecated, you should instead use: wsrep_cluster_address=gcomm://server_one,server_two,server_three iirc
    – Oneiroi
    Dec 6, 2012 at 15:22

Further research into this issue seems this is a viable method (leaving this answer unaccepted for the moment, incase someone replies with a better setup):

  1. Circular setup
    1. per pxc documentation have all nodes sync from node 1
    2. stop node2 repoint to node3, start node 2
    3. stop node3 repoint to node4, start node 3
    4. stop node1 repoint to node2, start node 1

This setup appears to tollerate the loss of any node via disconnection at least, and on restoration of the node syncs without issue.


If Mysql wont start and the reason is a corrupt DB table.

replicate what the server is doing and grab a good copy from a stopped server for client dbs.

it tars the files from $MYSQLHOME that are db via a nc.

we used scp to move the good files in place and then kicked off the sync again by starting mysql of the bad server.

  • Not sure how this help's the original poster.
    – slm
    Apr 6, 2013 at 11:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .