4

I am running a RabbitMQ cluster (version 3.0.2, Erlang R15B) with two nodes and they continue to periodically experience a network partition error. They are physically located in the same data center and their network should be reliable. When I check their logs, both servers report the “running_partitioned_network” error at about the same time and both nodes continue running, so I don’t think it is a hardware failure or one of the nodes terminating unexpectedly. I modified the net_ticktime to 120 seconds to try to mitigate the problem, and it stopped occurring for almost a month, but it recently started occurring again once every few days. Now I am not sure if the net_ticktime helped or if it was just coincidence.

In order to troubleshoot further, I started a rolling network trace using Wireshark and used a scheduled task to halt the trace when the nodes became partitioned again. My goal is to determine whether the partition is caused by unreliable network, or if the application failed to respond. Nothing in the packet trace jumps out as showing a network failure, there are only a handful of TCP retransmissions and plenty of other packets are sent successfully between them.

At this point I am not sure what else to look at in the packet trace to either prove or disprove that the network caused the failure. Wireshark can identify and decode the Erlang Distribution Protocol, but I don’t know how to interpret the messages to know what would cause nodes to detect a partition. Also, the net_ticktime is set to 120 seconds, and I do not see a 120 second gap in the servers receiving messages from each other. The longest gap in which no Erlang messages are received from the other server is 22 seconds (much less if you count the TCP acknowledgements). My only other thought is that if a particular “ping” type message needs to be sent between the nodes and that particular messages was interrupted, but I don’t know what that would look like in the trace.

Any ideas on how to further diagnose the cause of this issue would be helpful.

2 Answers 2

3

I'm not sure if this is really the case, but it seems Erlang clustering can break when big messages are being passed around. Take a look at this thread in the RabbitMQ discuss mailing list: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-March/018745.html

1
  • Thanks for the link! This is very interesting. I need to go back and look at my captures again, but this could explain why they detect a partition even though there is data passing back and forth. Still a little skeptical because my messages shouldn't be too long, but at least it is something to look at.
    – nslowes
    Aug 29, 2013 at 20:37
0

I've seen similar issues with RabbitMQ 2.8.4 on Erlang R14B03, though admittedly without the "running_partitioned_network" message. It hasn't happened in a few months for us (yes, it happened enough times that we have a Nagios check_rabbitmq_splitbrain check), but I'll see if I can capture some details if it happens again...

Not the answer you're looking for? Browse other questions tagged .