VM-1 on host-1 <[cable]> network router <[cable]> host-2 with VM-2
If I understand correctly, in case of file transfer from application on VM-1 to same application on VM-2 the data will go through following journey:
- VM-1 application file read to memory buffer
- programming language related calls
- operating system level calls
- seccomp/apparmor logic
- file system permissions logic
- operating system file handling and buffer
- VM-1 application data sent to network socket buffer
- operating system calls
- seccomp/apparmor logic
- VM-1 operating system network stack
- routing tables
- firewall logic
- Host-1 hypervisor virtual network stack
- virtual switch
- routing tables
- Host-1 operating system network stack
- routing tables
- firewall logic
- Host-1 physical network card buffer
- Network router
- almost same stack of things mirrored goes here for VM-2 on host-2
Assuming that file will be large, then steps related to seccomp/apparmor, routing and firewall will be cached/omitted for already openned and transfering file.
But in case of frequent communication between virtual machines with messages small enough to fit into 1-2 tcp packets we have problem
Every call and logic processing will need several hundred CPU ticks and described overstack will put significant load on CPU and play role in latency.
- As per Testing Docker multi-host network performance [August, 2016] it is at least -13% in performance.
- In Network I/O Latency on VMware vSphere® 5 "We found that on an idle system, round-trip latency overhead per VM is approx 15-20 microseconds. This number increases as the resource contention on vSphere increases, which is expected. Jitter was very low as long as the system was not overcommitted. "
- Additionally, Meltdown and Spectre Intel fix will result in even more performance drop.
Questions
- Will pre-openned communication socket between VMs ommit any steps in described list?
- Does SDN somehow mitigate such problems or does it add even more overlays and extra headers to packets?
- Do I really need described process to communicate between VM-1 and VM-2 or there is a special linux "less-secure-more-performance-use-on-your-own-risk" build?
- Do I have to stick with linux at all? Any faster *BSD-like systems with docker support?
- What are best practices to mitigate that bottleneck to fit more VMs with micro-services on same host as result?
- Do solutions like Project Calico help or it is more about lower level?