I have 3 servers networ kwise configured as follows
- A is a DELL R710 is running Linux 5.13.19-1-pve Proxmox VE 7.1 and has 4 NICs teamed in a
- B is a DELL R610 is running Linux 5.13.19-1-pve Proxmox VE 7.1 and has 4 NICs teamed in a
- C is a DELL R710 running FreeBSD 12.2-RELEASE-p1 with a lagg over 8 NICs in
roundrobin(this is a TrueNAS distro).
All NICs are 1 GBps.
When I run
iperf3 between the Linux blades, I max at about 3 GBps, and the window goes up to an average of ~300 KiB. However, between the TrueNAS (FreeBSD) blade and the Linux blades, the TCP stream maxes at 1.20 Gbps and caps the window at ~60 KiB average. If I run parallel streams (i.e.,
iperf3 ... -P 8) I can saturate the bond. On the other hand, as expected, the retransmit count is pretty high in both cases. So, my questions are,
- Why is FreeBSD not reaching the same throughput if supposedly both are approaching the problem in the same way? (maybe that's where I am wrong).
- Is there a tuning option or combination of options to make the TCP stack more tolerant to out-of-order without triggering immediate retransmits? (I am vaguely familiar with the 3-ACK reTX, basics of TCP congestion control, and so on).
I will include here some tunables and options I have used during my testing.
- All ifaces are set to use jumbo frames (MTU 9000).
- The Linux boxes are tuned as follows
net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 net.ipv4.tcp_mem = 1638400 1638400 1638400 net.ipv4.tcp_rmem = 10240 87380 16777216 net.ipv4.tcp_rmem = 10240 87380 16777216 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_reordering = 127 net.ipv4.tcp_max_reordering = 1000 net.core.netdev_max_backlog = 10000 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.tcp_mtu_probing = 1 net.ipv4.tcp_congestion_control = reno
- The FreeBSD (TrueNAS Core ~= FreeNAS) box is tuned as follows
kern.ipc.maxsockbuf=614400000 kern.ipc.somaxconn=1024 net.route.netisr_maxqlen=8192 net.inet.ip.intr_queue_maxlen=8192 net.inet.tcp.mssdflt=8948 net.inet.tcp.reass.maxqueuelen=1000 net.inet.tcp.recvbuf_inc=65536 net.inet.tcp.sendbuf_inc=65536 net.inet.tcp.sendbuf_max=307200000 net.inet.tcp.recvbuf_max=307200000 net.inet.tcp.recvspace=65228 net.inet.tcp.sendspace=65228 net.inet.tcp.minmss=536 net.inet.tcp.abc_l_var=52 net.inet.tcp.initcwnd_segments=52 # start fast net.inet.udp.recvspace=1048576 net.inet.udp.sendspace=1048576