I'm connecting several KVM VMs to a virtual network that is routed to a 1Gbit physical network. The router uses netfilter/iptables to filter traffic between the real and virtual networks. For the virtual network switch I'm using SR-IOV with PCI-passthrough. Compared to using a Linux bridge, this setup allows higher throughput (limited by PCIe bandwidth) with lower CPU overhead (ref: pp 22-23 of Toshiaki Makita's presentation at LinuxCon Japan, 2014.
I've assigned one VF of the same port of an Intel I350 NIC to each VM and to the KVM host. (Each I350 port has a maximum 7 VFs, so 6 VMs + host is the maximum size of this virtual network). This setup is working as expected except for an annoying quirk: the I350's embedded switch only functions when I connect the associated I350 physical port with a physical patch cable to a physical external switch (which has all other ports empty). With the external switch on, the virtual network works fine; but when the external switch is powered down the PF link status changes to "NO-CARRIER" and the virtual network no longer passes packets.
Does anyone know of a way to get the I350's embedded switch to function without an active link on the physical port?
The VM host runs Debian 10 (Buster), if that matters.
Thanks for any light you can shed!
A loopback plug like the Smartronix SuperLooper would be less clunky than an external switch, and might work. However according to the manufacturer it's "intended solely for testing systems where the Near End Crosstalk (NEXT) function can be disabled" -- and the I350 datasheet does not mention such functionality.
Section 3.7.6 of the I350 datasheet describes 4 different supported internal loopback modes. (The same internal loopback functionality is also present in the I210, and likely other Intel chips as well.) However I have not found any information on using Linux tools to configure the I350/I210/etc to use internal loopback. It's also unclear whether activating one of the internal loopback modes would also activate the I350's embedded switch...?
Thanks to @Tomek, I tried
# ip link set dev eth1 vf 0 state enable RTNETLINK answers: Operation not supported
ip link set dev eth1 vf 0 trust ondoes work, so syntax is correct and driver is working. I got curious whether the igb driver or I350 hardware prevents setting vf link state to
enable. Looking at i40e_main.c (for example),
struct net_device_ops i40e_netdev_opscontains
.ndo_set_vf_link_state = i40e_ndo_set_vf_link_state, .ndo_set_vf_spoofchk = i40e_ndo_set_vf_spoofchk, .ndo_set_vf_trust = i40e_ndo_set_vf_trust,
whereas in igb_main.c
struct net_device_ops igb_netdev_ophas
.ndo_set_vf_spoofchk = igb_ndo_set_vf_spoofchk, .ndo_set_vf_trust = igb_ndo_set_vf_trust,
but is missing
.ndo_set_vf_link_state. So it looks like
igbdoes not support setting VF link state to enabled. Whether the I350 hardware could support this feature is another question. It seems this standard way of enabling the embedded switch will not work for I350. Maybe there's another way?
The I350 datasheet makes some discouraging statements:
220.127.116.11 Packet Switching (VMDq) Model: VMDq Assumptions
- When the link is down, the Tx flow is stopped, and thus the local switching traffic is stopped also.
18.104.22.168 TX Packets Switching
The following rules apply to loopback traffic:
- Loopback is disabled when the network link is disconnected.
Whereas the Intel 710 datasheet reads quite differently:
Table 1-7. Internal Switching Features
Internal switching operates independently of the state of the LAN ports (also when LAN ports are down).
It's increasingly looking like the answer to my question is: Yes, an I350 does need an external switch attached in order to switch VM-VM traffic. I'd love for someone to prove me wrong!