Hi ServerFault Community!
I have deployed a Rancher 2.0 K8s Cluster with 5 Nodes which have 2 network interfaces (ens3 with Internet Public IP, and ens4 with datacenter private network). Each of the nodes is registered in Rancher with both IP correctly defined. This cluster uses Canal as network provider and the private network is indeed twice as fast as the public network, so it is definitely preferable for all inter-server communications.
The thing is, most of multi-tier App Deployment strategies is to put (MySQL for ex) behind a Headless Service and let the App Tier (Wordpress for ex) access the DB through this service, by resolving the DNS name of the DB Service.
So far, so good, until I adventured to check through which network interface this particular Pod to Pod traffic went through... Effectively, it was going through the public network interface, and unencrypted! Just an image below:
As you can easily appreciate, I was testing with ping from the App tier pod to the service, and the ping content is easily readable in this UDP packet flowing through the public interface.
Questions pending for me, and hopingly this awesome community can shed some light on some of it: I do not really know if this traffic is anything related to Canal, but I bet on it. So the encryption topic is not applicable here, but at least, there is some way to make it use the private network? I already tried to change cluster configuration file accordingly with:
network: canal_network_provider: iface: "ens4" options: flannel_backend_type: "vxlan" plugin: "canal"
And now I think I see some more sources of traffic through the private interface. (Hard to distinguish, as the GlusterFS running there is so chatty...)
But to no avail, the shown Wireshark sample was taken after these changes, and the Pod<->Pod service traffic continues to be done on top of the public network.
So, Ideal situation would be the one in which all traffic except Ingresses run on the private network, thus not needing encryption and enjoying far better connection infra.
¿Any suggestions on how to achieve this? I have no problem in switching from Rancher to other K8s provider, as long as it is free and supports bare-metal deployments. Also another CNI plugin could be used, but I am yet in doubt if this traffic is managed by the CNI or not...
Thank you great ServerFault community!