5

I have tried every conceivable combination in an attempt to cluster rabbitmq in an AWS environment. But to recap:

  1. Shutdown and removed erlang and rabbit distribution on my local ubuntu 14
  2. Tried the auto configuration modules around the web
  3. The ubuntu 14 version the comes default installed won't cut it.
  4. The erlang cookies match - this is demonstrated below

The hostname mismatch is the only puzzle. The node itself thinks it's hostname is 'q1' or 'q2' respectively. When I try to set the host name of the container to the private dns name of the host (so it can connect to the other node) the rabbit instance in the container crashes. Not below how hostname produces q2 but I shelled into an amazon private dns?

root@q2:~# hostname
q2
root@q2:~# exit
christian@q2:~$ logout
Connection to ip-10-0-3-101.us-west-2.compute.internal closed.

I am using the latest rabbitmq docker image.

docker run -d --restart always --hostname q1 --name rabbitmq -p 4369:4369 -p 15671:15671 -p 25672:25672 -p 15672:15672 -p 5672:5672 -e RABBITMQ_HIPE_COMPILE=1 -e RABBITMQ_ERLANG_COOKIE='ilikecookies' rabbitmq:3-management

The service starts up just fine

root@q1:~# curl -I localhost:15672
HTTP/1.1 200 OK
Content-Length: 1419
Content-Type: text/html
Date: Fri, 20 Jan 2017 22:46:12 GMT
last-modified: Fri, 20 Jan 2017 22:38:45 GMT
Server: MochiWeb/1.0 (Any of you quaids got a smint?)

And here is the cookie from host q1

root@q1:~# docker exec -it rabbitmq /bin/bash
root@q1:/# cat /var/lib/rabbitmq/.erlang.cookie                                                                               
ilikecookies
root@q1:/# 

Now I attempt to cluster it (from host q2 with q1 being the master)

root@q2:~# docker exec -it rabbitmq /bin/bash
root@q2:/# rabbitmqctl stop_app
Stopping node rabbit@q2 ...
root@q2:/# rabbitmqctl join_cluster rabbit@ip-10-0-3-56.us-west-2.compute.internal
Clustering node rabbit@q2 with 'rabbit@ip-10-0-3-56.us-west-2.compute.internal' ...
Error: unable to connect to nodes ['rabbit@ip-10-0-3-56.us-west-2.compute.internal']: nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@ip-10-0-3-56.us-west-2.compute.internal']

rabbit@ip-10-0-3-56.us-west-2.compute.internal:
  * connected to epmd (port 4369) on ip-10-0-3-56.us-west-2.compute.internal
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?
  * suggestion: is the Erlang distribution using TLS?

current node details:
- node name: 'rabbitmq-cli-41@q2'
- home dir: /var/lib/rabbitmq
- cookie hash: quN0y0GUm2Zxv8VYc2eX9A==

root@q2:/# cat /var/lib/rabbitmq/.erlang.cookie
ilikecookies
root@q2:/# 

The question is HOW do you get these things to cluster? What's the missing ingredient? The error message means nothing around the web. Does anyone have some experience in this?

UPDATE AWS security group for these instances:

Custom TCP Rule
TCP
1024 - 65535
0.0.0.0/0
  • 1
    "TCP connection succeeded but Erlang distribution failed". There are a few suggestions there. Have you considered those? – Tim Jan 21 '17 at 2:19
  • Such obvious questions. The problem is that response is so common it doesn't narrow a Google search down – Christian Bongiorno Jan 21 '17 at 2:32
  • 1
    Generally you should avoid trying to insult people trying to help you, especially when you haven't described what you've done to try to solve the problem yourself. Problem solving is often a process of elimination. Good luck. – Tim Jan 21 '17 at 3:13
  • Sorry if you took offense. But, those were things I looked at already. Perhaps I should make that clear. The cookie issue is already addressed in the question – Christian Bongiorno Jan 21 '17 at 3:18
  • Perhaps you should add what you've tried to solve the problem to your question. – Tim Jan 21 '17 at 3:50
2

OK, I got it!

the host names of the each node have to line up inside the container.

On the host machine (q2) I checked what hosts it knew of in the hosts file:

# This file was generated by OpsWorks
# any manual changes will be removed on the next update.

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

# OpsWorks Layer State
127.0.0.1 localhost.localdomain localhost
127.0.1.1 q2.localdomain q2

10.0.3.56 q1.localdomain q1
10.0.3.101 q2.localdomain q2


root@q2:/# ping q1
PING q1.local (10.0.3.56): 56 data bytes
^C--- q1.local ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

Then,it occurred to me that it doesn't matter at all what the host machine knows, it's what the docker container knows about. So, I shelled into the container and did the same thing:

root@q2:/# cat /etc/hosts
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2  q2.local

Now we are on to something! So, I added an entry for the node master:

root@q2:/# echo "10.0.3.56    q1.local q1" >> /etc/hosts
root@q2:/# which ping
/bin/ping
root@q2:/# ping q1
PING q1.local (10.0.3.56): 56 data bytes

And, took another swing inside the container

root@q2:/# rabbitmqctl stop_app
Stopping node rabbit@q2 ...
root@q2:/# rabbitmqctl join_cluster rabbit@q1                                     
Clustering node rabbit@q2 with rabbit@q1 ...
root@q2:/#

And now, each node recognizes that it's clustered! Woot!

I think for clustering with docker I am going to modify the docker command to mount the hosts /etc/hosts file into the docker image with -v /etc/hosts:/etc/hosts:ro and then this should just work magically

Another step I forgot to mention: The local Ubuntu box had an ancient version of erlang running that I had to remove (and it had rabbit as well).

0

This project automates the clustering process in AWS ECS. It should make the process less painful and more resilient to failures.
https://github.com/malawson/rabbitmq-ecs-autoclustering

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.