So, this is an interesting problem that perhaps can be shed some light on. I'm running RabbitMQ 3.0.1 (Installed via Brew) on OS X Mountain Lion 10.8 in ZSH with Oh-My-ZSH.
The issue is, I create a cluster locally (all 6 nodes running on my one computer), and then turn some of them on and off, simulating nodes going down (with stop_app and start_app). At one point, I shut down all of the nodes.
Then, I bring the last one shut down back online. It comes up. I bring another one online, it connects to the cluster.
I then shut them both down again.
When I bring back the last one shut down... it fails to start. From here on out, I cannot get the nodes to restart. Why?
$ ra -n r_hare0 cluster_status
Cluster status of node r_hare0@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander,r_hare1@Charmander,r_hare2@Charmander,
r_hare3@Charmander,r_hare4@Charmander,r_hare5@Charmander]}]},
{running_nodes,[r_hare2@Charmander,r_hare3@Charmander,r_hare0@Charmander]},
{partitions,[]}]
...done.
As you can see here, I have 6 nodes running - at this point only 3 of them are on. All of my nodes ahve been started in disc mode - none are "ram".
$ ra -n r_hare2 stop_app
Stopping node r_hare2@Charmander ...
...done.
$ ra -n r_hare3 stop_app
Stopping node r_hare3@Charmander ...
...done.
$ ra -n r_hare0 cluster_status
Cluster status of node r_hare0@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander,r_hare1@Charmander,r_hare2@Charmander,
r_hare3@Charmander,r_hare4@Charmander,r_hare5@Charmander]}]},
{running_nodes,[r_hare0@Charmander]},
{partitions,[]}]
...done.
Only 1 node is left running, let's turn it off, simulating the cluster shutting down.
$ ra -n r_hare0 stop_app
Stopping node r_hare0@Charmander ...
...done.
$ ra -n r_hare0 cluster_status
Cluster status of node r_hare0@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander,r_hare1@Charmander,r_hare2@Charmander,
r_hare3@Charmander,r_hare4@Charmander,r_hare5@Charmander]}]}]
...done.
Good - the cluster is down.
$ ra -n r_hare1 start_app
Starting node r_hare1@Charmander ...
BOOT FAILED
===========
Error description:
{could_not_start,rabbit,
{bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_user_permission,rabbit_vhost,
rabbit_durable_route,rabbit_durable_exchange,
rabbit_runtime_parameters,
rabbit_durable_queue]}}}}}}}
Log files (may contain more information):
/usr/local/var/log/rabbitmq/r_hare1.log
/usr/local/var/log/rabbitmq/r_hare1-sasl.log
Error: {rabbit,failure_during_boot,
{could_not_start,rabbit,
{bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_user_permission,
rabbit_vhost,rabbit_durable_route,
rabbit_durable_exchange,
rabbit_runtime_parameters,
rabbit_durable_queue]}}}}}}}}
This is expected. This was not the last app down, so it cannot start up. Good! Let's restart the correct node.
$ ra -n r_hare0 start_app
Starting node r_hare0@Charmander ...
...done.
Success!
$ ra -n r_hare1 start_app
Starting node r_hare1@Charmander ...
...done.
Fantastic.
$ ra -n r_hare0 cluster_status
Cluster status of node r_hare0@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander,r_hare1@Charmander,r_hare2@Charmander,
r_hare3@Charmander,r_hare4@Charmander,r_hare5@Charmander]}]},
{running_nodes,[r_hare1@Charmander,r_hare0@Charmander]},
{partitions,[]}]
...done.
Everything looks good. OH GOD AMAZON GOES DOWN AGAIN
$ ra -n r_hare0 stop_app
Stopping node r_hare0@Charmander ...
...done.
$ ra -n r_hare1 stop_app
Stopping node r_hare1@Charmander ...
...done.
This time, hare1 is the last one down.
$ ra -n r_hare0 cluster_status
Cluster status of node r_hare0@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander,r_hare1@Charmander,r_hare2@Charmander,
r_hare3@Charmander,r_hare4@Charmander,r_hare5@Charmander]}]}]
...done.
So let's start Hare1 back up again and keep moving...
$ ra -n r_hare1 start_app
Starting node r_hare1@Charmander ...
BOOT FAILED
===========
Error description:
{could_not_start,rabbit,
{bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_user_permission,rabbit_vhost,
rabbit_durable_route,rabbit_durable_exchange,
rabbit_runtime_parameters,
rabbit_durable_queue]}}}}}}}
Log files (may contain more information):
/usr/local/var/log/rabbitmq/r_hare1.log
/usr/local/var/log/rabbitmq/r_hare1-sasl.log
Error: {rabbit,failure_during_boot,
{could_not_start,rabbit,
{bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_user_permission,
rabbit_vhost,rabbit_durable_route,
rabbit_durable_exchange,
rabbit_runtime_parameters,
rabbit_durable_queue]}}}}}}}}
Wait. What?
$ ra -n r_hare0 start_app
Starting node r_hare0@Charmander ...
BOOT FAILED
===========
Error description:
{could_not_start,rabbit,
{bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_user_permission,rabbit_vhost,
rabbit_durable_route,rabbit_durable_exchange,
rabbit_runtime_parameters,
rabbit_durable_queue]}}}}}}}
Log files (may contain more information):
/usr/local/var/log/rabbitmq/r_hare0.log
/usr/local/var/log/rabbitmq/r_hare0-sasl.log
Error: {rabbit,failure_during_boot,
{could_not_start,rabbit,
{bad_return,
{{rabbit,start,[normal,[]]},
{'EXIT',
{rabbit,failure_during_boot,
{error,
{timeout_waiting_for_tables,
[rabbit_user,rabbit_user_permission,
rabbit_vhost,rabbit_durable_route,
rabbit_durable_exchange,
rabbit_runtime_parameters,
rabbit_durable_queue]}}}}}}}}
Erm... these were the last two nodes down... and yet neither one of them can restart.
$ ra -n r_hare0 cluster_status
Cluster status of node r_hare0@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander]},
{ram,[r_hare1@Charmander,r_hare2@Charmander,r_hare3@Charmander,
r_hare4@Charmander,r_hare5@Charmander]}]}]
...done.
What the Hell happened to my cluster? Why does Hare0 think there are Ram nodes in the cluster? Why does it think it's the only disc node?
$ ra -n r_hare4 cluster_status
Cluster status of node r_hare4@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander,r_hare1@Charmander,r_hare2@Charmander,
r_hare3@Charmander,r_hare4@Charmander,r_hare5@Charmander]}]}]
...done.
Hare4 disagrees.
$ ra -n r_hare1 cluster_status
Cluster status of node r_hare1@Charmander ...
[{nodes,[{disc,[r_hare1@Charmander]},
{ram,[r_hare0@Charmander,r_hare2@Charmander,r_hare3@Charmander,
r_hare4@Charmander,r_hare5@Charmander]}]}]
...done.
But Hare1, which was shutdown last, thinks it is the only disc mode node.
$ ra -n r_hare3 cluster_status
Cluster status of node r_hare3@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander,r_hare1@Charmander,r_hare2@Charmander,
r_hare3@Charmander,r_hare4@Charmander,r_hare5@Charmander]}]}]
...done.
$ ra -n r_hare2 cluster_status
Cluster status of node r_hare2@Charmander ...
[{nodes,[{disc,[r_hare0@Charmander,r_hare1@Charmander,r_hare2@Charmander,
r_hare3@Charmander,r_hare4@Charmander,r_hare5@Charmander]}]}]
...done.
Now everyone is confused.
