I am working on a production setup with Docker in swarm mode (using Docker 17.03.1-ce). There will be 2 data centers involved. In both data centers, all machines have both a public IP and a private (local to data center) IP on a private network, so there will be 2 private networks.

Link to simplified diagram visualizing setup

Network traffic over the private network interfaces is free, while traffic over the public interface isn't after some limit (and it's slower), so where possible I would prefer network traffic to go over the private interfaces.

Now from what I understand (I think), all traffic between Docker nodes in swarm mode will go through the same network interface that's used for communicating with the swarm masters, which in my case will have to be the public ones to make multi DC networking possible. Most expected traffic however will be between nodes in the same DCs, and it would be very nice if Docker could somehow route traffic through my private networks if source and target nodes happen to be on the same private network.

I'm afraid this might not be possible out of the box because the swarm masters don't know about these private networks and the IPs the nodes have on them.

One solution I could think of is to set up a VPN and deploy the swarm on top of that, but this adds extra complexity and I'd prefer a pure Docker swarm solution.

Update: as suggested in a comment, the basis for a solution might be using iptables to route outgoing traffic to private IPs instead of public ones. However, if I would want to do this, my next problem would be how to manage all these rules. With 10 servers in a DC, I'd need 10 * 9 = 90 of them to route all the possible local traffic over the private network. I can imagine maybe some tool exists that could assist with such a task or I could create one, but maybe there's a much simpler way to do this.

migrated from stackoverflow.com Apr 9 '17 at 19:52

This question came from our site for professional and enthusiast programmers.

  • 1
    You could use iptables to rewrite the destination ip of siblings in same datacenter to their private network ips. Also, this should be on serverfault not SO ;) – n00b32 Apr 7 '17 at 9:23
  • Yeah, some more digging led me to the same conclusion. That's going to be a hassle though, I'd really need a tool to do that for me automatically (rewrite all IPs between all nodes in a cluster/DC). After all, for N nodes I'd need (N-1)^2 iptables rules. – Ede Apr 7 '17 at 11:18

(Probably it should be a comment, but cannot comment yet)

As you cannot use hostnames and/or DNS to bootstrap the cluster, I see no way to force the cluster exchange raft data on the correct unmetered network, but I see you can use an interface name. Curiously, it is not stated on the swarm docs but an issue show the error message expects an IP or interface.

I wonder if you can set the cluster members to advertise using the private interface name, so that you get the most traffic in a the way you want.

Cannot test myself right now, but will next week as I may run into a similar issue for a upcoming project.

  • Thanks, please let me know if you find out anything next week. The issue you linked to is about the 'old' Docker swarm functionality, whereas I'm using (and would advise you to use) the new Docker swarm mode which is built-in and doesn't require an external key/value store. – Ede Jun 13 '17 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.