2

I have been experimenting with Pacemaker & Corosync for a HA setup. I installed it on two Ubuntu 14.04 Linode's and after a lot of back and forth on outdated config examples and tutorials, managed to get it up and running. This mostly due to the fact that I had to set it to use UDP Unicast for it to work(Multicast and Broadcast won't work on Linode's network)

Both nodes are able to see each other, and they detect when the other one goes down. Great!

Now I want to try to add a third node while the cluster is running, so I go hunting for some command to do so... No...Nope... This seemingly crucial and simple action is, as far as I can tell, not possible with a simple command.
To add a node you have to add it to the config of every single node, and then shut down the cluster, reload, and start it back up. There goes High-Availability out of the window.

The only way to dynamically add node that I could find, was under cmap_keys(8).
I have to first run a command to find the next free node id. Then, for each node currently in the cluster, manually add it to nodelist.node.x.ringy_addr, and so on. Even after all that it will not survive a restart, I also have to update each and every config file, manually. THEN I have to copy the new config file to the new node, and start it up.

It is possible to connect to any of the nodes in the cluster, give a command to change existing nodes, and add resources, and these will be synced between all nodes. But I can't simply add a new node in the same way?

Please, someone tell me I'm just bad at googling information, and there is a simple command to add new nodes in a live, running HA cluster. Is there some tool or method I have overlooked?

Thank you for your time!

1

If you use multicast you can quickly and easily add new nodes to the cluster. Because of this I suspect that the developers may have never saw a need to allow dynamically adding nodes via UDP unicast. At least no "clean" way exist that I myself am aware of.

However, you should be able to reload Corosync and apply the newly edited config files without interrupting services by first putting Pacemaker into maintenance-mode. When in maintenance-mode none of the currently running services will be stopped when you stop and restart Corosync and Pacemaker.

  • Thank you for this =) Got a good night sleep and then took a look at maintenance mode as you suggest. While not the solution I want, it seems workable. Use a tool like Chef to push new config to all nodes. Put into maintenance mode. Restart and make sure everyone connects ok. Go out of maintenance mode. Should only have a minute or so downtime window, where it would not do failover. What happens if something goes down while I'm in maintenance mode? Will it detect and handle it once I go out of maintenance mode? – Wildex999 Jul 16 '14 at 23:20
  • I also notice that I can add a new node through an "introducer" node. As long as the introducer and the new node know about each other, the new node will propagate to the other nodes. However, if the introducer node is down, and the new node goes down, the others can't detect it coming back. Probably bad in case of split brain etc. – Wildex999 Jul 16 '14 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.