We have a RedHat 5.8 VM running on ESX on top of Cisco UCS that is acting as a firewall using iptables. The box has multiple nics, one nic serves as the gateway to our network with the other nics each going to a separate VLAN/network. The problem I'm seeing is that doing an iperf test from a host directly to the Firewall VM gives 800+ Mbps speed but trying to do an iperf test from a host outside the firewall to a host inside the firewall (causing the traffic to pass through the firewall) yields speeds of about 30Mbps or less.

The details:

  • I'm not sure what release or versions of ESX or UCS we are running, but I can find out if that's important. Our sysadmin keeps it up to date so they're most likely within the last couple of updates.
  • I know we are using the VMXNET 3 driver for the network adapters.
  • All connections are verified to be 1Gbps.

What I've tried:

  • From what I had read, the e1000 driver has better results, so we added a couple of nics using that driver and iperf tested through those interfaces with the same results.
  • Checked that LRO was disabled on the tested interfaces. When I ran 'ethtool -K ethX lro off', it reports 'no offload settings changed' which I assume to mean it's already disabled.
  • Also disabled TSO on the tested interfaces.
  • When I did my speed tests, I was typically testing from a physical device to a virtual device through the firewall on the same cluster. I also tried testing from a virtual device to a virtual device through the firewall and got the same results.
  • Disabled iptables and ran speed tests, receiving the same results.
  • None of the items above changed anything except possibly making things even slower (I got <10 Mbps at one point).

Because I can get 800+ Mbps going to the firewall itself, that leads me to think that there is nothing wrong with the configuration of the nics themselves. I feel like it's some issue with the forwarding on the OS itself since it only seems to manifest when it's passing traffic through the firewall. I should also note that I didn't observe any CPU spikes during this process.

I'm sure I'm leaving out some details, so if there are any further questions, let me know. I appreciate any help!


We found the solution. We had to disable LRO at the blade level on the UCS (as shown here -> http://docwiki.cisco.com/wiki/Disable_LRO). Once we did that, the speeds bumped up to 700+MBps, limited only because of the CPU and iptables processing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.