We have a Linux firewall with two outward facing 10Gbe adapters (Intel 82599EB) and one inward facing 10Gbe adapter (Intel 82598EB).
The problem I'm experiencing is that the firewall will only forward inbound traffic at a very low rate: approximately < 2 Mbps. However, a direct connection from the firewall to an "inside" machine gets ~6 Gbps, while a direct connection to the firewall from an outside machine gets ~1 Gbps. There's some tuning to be done clearly, but they demonstrate Gbps speeds.
We recently updated the Intel
ixgbe driver from version 2.1.4 to 3.7.14 due to stability concerns with the 2.1.4 driver (lock-ups) and this seems to be when the throughput problems began.
I also tried the 3.7.17 release, but this gave similar performance to 3.7.14. On reverting to the 2.1.4 driver (re-compiled for an updated kernel, with IXGBE_NO_LRO and IXGBE_NO_NAPI) I was able to get ~Gbps throughput (well ~900 Mbps with iperf over TCP with 3 threads).
This solves the immediate problem, but I would prefer to be able to use the current version of the driver as I'd like to keep up with bug-fixes etc. so, my question is
- How can I troubleshoot Linux router/firewall forwarding performance?
Specifically, how can I find out where the kernel / iptables / network driver, etc. are spending their time when forwarding packets?
Any relevant advice would be appreciated.