Mixed Riak cluster with docker container instances and non-container instances

Question

I have a riak cluster that is running on multiple virtual machines scattered about. I am now experimenting with docker and would like to add nodes to the cluster that are running within docker containers.

How do I configure the networking components in this scenario? My containers can see the non-container instances and can join the cluster but the non-containerized instances can't initiate communications with the containerized instances.

Let's say I have two VMs running riak with IP addresses of 192.168.1.1 and .2. My docker host has an IP address of 192.168.1.3. Three riak containers are assigned IP addresses of 172.17.0.1, .2, and .3. I joined each container instance to the cluster.

Running riak-admin member-status on any node shows something like so:

Status   Ring   Pending  Node
------------------------------------------
valid     0.0%    32.8%  'riak@172.17.0.1'
valid     0.0%     0.0%  'riak@172.17.0.2'
valid     0.0%     0.0%  'riak@172.17.0.3'
valid    50.0%    32.8%  'riak@192.168.1.1'
valid    50.0%    32.8%  'riak@192.168.1.2'

The cluster can't balance itself because the VMs can't initiate comms with the containers.

I know how to expose ports in my container and publish those on the docker host but I'm not sure how I would then join a containerized riak node to the cluster using the docker hosts IP address and published port...

Answer 1

Multi-datacenter replication is for copying data between clusters, it probably won't help set up your cluster.

The key will likely be opening all of the ports that are needed, which include:

tcp/4369 (epmd)
tcp/8098 (riak http)
tcp/8087 (riak protocol buffers)
tcp/8099 (riak handoff)
tcp/x    (riak disterl)

The disterl port is chosen from the OS ephemeral port range by default, but you can restrict it using the erlang.distribution.port_range.minimum and erlang.distribution.port_range.maximum settings.

Ports 8098 and 8087 are needed for clients to reach the Riak instance in the container.

Port 4369 is for EPMD which is the port mapper that allows other nodes to find the disterl port of the Riak node.

Port 8099 is needed for the Riak instance to receive handoff from other nodes.

The disterl port is the primary communication channel between nodes.

4369, 8099 and the disterl port need to be exposed in order for the cluster to function, the other two are only needed if client requests will directly reach the instances in the containers.