4

I run RabbitMQ on 3 servers, same version of Erlang and RabbitMQ: RabbitMQ 3.4.1, Erlang 17.3

One node crashed on server 2. The two other nodes did not connect together:

server 1:

[CentOS-62-64-minimal ~]$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@CentOS-62-64-minimal' ...
[{nodes,[{disc,['rabbit@CentOS-62-64-minimal',rabbit@de3,rabbit@mysql]}]},
 {running_nodes,['rabbit@CentOS-62-64-minimal']},
 {cluster_name,<<"rabbit@CentOS-62-64-minimal">>},
 {partitions,[]}]

server 3:

[de3 ~]$ sudo rabbitmqctl cluster_status
Cluster status of node rabbit@de3 ...
[{nodes,[{disc,['rabbit@CentOS-62-64-minimal',rabbit@de3,rabbit@mysql]}]},
 {running_nodes,[rabbit@de3]},
 {cluster_name,<<"rabbit@CentOS-62-64-minimal">>},
 {partitions,[]}]

After restarting and resetting rabbitmq on server 3, it finally connected to server1:

[CentOS-62-64-minimal ~]$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@CentOS-62-64-minimal' ...
[{nodes,[{disc,['rabbit@CentOS-62-64-minimal',rabbit@de3,rabbit@mysql]}]},
 {running_nodes,['rabbit@CentOS-62-64-minimal']},
 {cluster_name,<<"rabbit@CentOS-62-64-minimal">>},
 {partitions,[]}]

Why did the cluster "break" with just 1 node down? server 3 was working fine, but server 1 was not: "Queue is located on a server that is down".

As for server 2, it did not restart. After a manual restart, I cannot make it reconnect to the cluster, even after multiple reset and removing /var/lib/rabbitmq/mnesia/:

[root@mysql ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@mysql ...
[{nodes,[{disc,[rabbit@mysql]}]},
 {running_nodes,[rabbit@mysql]},
 {cluster_name,<<"rabbit@mysql.domain.com">>},
 {partitions,[]}]

[mysql ~]# rabbitmqctl stop_app
Stopping node rabbit@mysql ...
[root@mysql ~]# rabbitmqctl force_reset
Forcefully resetting node rabbit@mysql ...
[ysql ~]# rabbitmqctl join_cluster rabbit@CentOS-62-64-minimal
Clustering node rabbit@mysql with 'rabbit@CentOS-62-64-minimal' ...
Error: {ok,already_member}
[mysql ~]# rabbitmqctl start_app
Starting node rabbit@mysql ...
[mysql ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@mysql ...
[{nodes,[{disc,[rabbit@mysql]}]},
 {running_nodes,[rabbit@mysql]},
 {cluster_name,<<"rabbit@mysql.domain.com">>},
 {partitions,[]}]

I have no idea what went wrong. Last time this happened, I upgraded RabbitMQ qnd Erlang to the latest version.

  • As a starting point, check your logs for clues /var/log/rabbitmq (and post the relevant sections). Also, you can try: rabbitmqctl forget_cluster_node rabbit@mysql from server 1 or 3 (the nodes that are still part of the cluster) before attempting to add server 2 back to the cluster. – cyberx86 Nov 21 '14 at 6:06
1

I had this issue today designing an intentional break document for a breakfix event to teach our operations team how to fix stuff. I intentionally unclustered a node and was unable to run the rabbitmqctl join_cluster successfully because the cluster believed the node to already be a member.

Clustering node 'rabbit@node-1' with 'rabbit@node-0' ... ...done (already_member).

Ultimately what worked for me was rabbitmqctl forget_cluster_node rabbit@node-1 from a working clustered node. Once I did that, I was able to successfully run rabbtmqctl join_cluster rabbit@node-0

4

Based on the RabbitMQ cluster documentation your rabbitmqctl cluster_status output looks wrong; running_nodes should contain more than just the local node where you are running the command. That suggests to me that they can't talk to each other properly, are there any firewalls in between the nodes?

  • Checking if DNS entries have changed might be worthwhile, too. – Markus W Mahlberg Nov 21 '14 at 14:22
1

Bodgit is correct, I can tell you from having an operational rabbit cluster that your configuration is wrong. It looks like each node is it's own cluster with only itself as the current running node.

Please refer back to the RabbitMQ doc on setting up the cluster.

You should see something much more like the following on each node:

    root@rabbit0:~# rabbitmqctl cluster_status
    Cluster status of node 'rabbit@rabbit0' ...
    [{nodes,[{disc,['rabbit@rabbit0','rabbit@rabbit1']}]},
     {running_nodes,['rabbit@rabbit1','rabbit@rabbit0']},
     {cluster_name,<<"rabbit@rabbit0.domain.com">>},
     {partitions,[]}]
    ...done.

    root@rabbit1:~# rabbitmqctl cluster_status
    Cluster status of node 'rabbit@rabbit1' ...
    [{nodes,[{disc,['rabbit@rabbit0','rabbit@rabbit1']}]},
     {running_nodes,['rabbit@rabbit0','rabbit@rabbit1']},
     {cluster_name,<<"rabbit@rabbit0.domain.com">>},
     {partitions,[]}]
    ...done.

This is sanitized but the orders and intent is kept.

You also need to configure high availability if you want your queues to fail over:

https://www.rabbitmq.com/ha.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.