2

I am trying to use an old Fujitsi RX300S2, with a quad core Intel Xeon CPU @2.80GHz as a Gitabit NAT router, it has a dual gigabit NIC on board over PCI-X.

The router will also forward multicast traffic from the external interface to the internal network. Multicast routing is handled by the upstream Cisco router so the NAT router only has to "leak" multicast traffic between eth1 (upstream) and eth0 (internal).

This has been properly setup using igmpproxy which basically makes the L3 router act as a L2 bridge according to multicast traffic.

When testing the throughput, I have no problem receiving ~850-900Mbit multicast traffic on 200 groups/streams (approx 80'000 p/s) to a local process in userspace, which also analyses the 200 streams in realtime without packet loss. The local process maxes one core at 100%.

The streams consists of IPTV mpeg transport streams encapsulated in IP UDP packets. 7x188=1316 bytes of payload.

But when testing the throughput in forwarding mode, e.g multicast traffic enters eth1 and is routed at kernel level to eth0 and sent into the local network, the NAT router cannot forward all traffic received.

The external interface eth1 receives all multicast traffic ~900Mbit but the outgoing interface only transmits ~600Mbit and all streams suffer from packet loss according the the receiving test machine attached to eth0.

When analysing the load ksoftirqd/3 maxes out at 100% CPU but the other 3 cores are below 10% so it seems that not all 4 cores participate in the load.

The /proc/interrupts also shows that eth0 and eth1 share irq16:

    CPU0 CPU1  CPU2       CPU3
16:    0    0 92155  208280892   IO-APIC   16-fasteoi uhci_hcd:usb2, uhci_hcd:usb5, eth1, eth0

As can be seen, CPU3 handles a disproportionate amount of interrupts.

I have read through various texts regarding cpu_affinity and trying to pin CPU cores to network queues. Unfortunately this NIC tg3 from Broadcom does not support multiple queues but still, it should be possible to share the load between more cores on this quad core system.

Or is it the PCI-X bus that is the bottleneck, but if so, then the throughput should be reduced on both incoming eth1 and outgoing eth0 and packets should be dropped by eth1 but it seems that packets are lost between eth1 and eth0. Not true, since when packets are lost in the router, /sys/class/net/eth1/statistics/rx_missed_errors is incremented alot (about 1000 p/s).

When only 100 channels and approx 500 Mbit is forwarded, packet loss does not happen and ksoftirqd/3 only consumes about 5-6% CPU. But when 600Mbit is forwarded, ksoftirqd/3 consumes 100% so it seems that some bottleneck outside the CPU is hit.

Is it out of the question that an old server like this is able to forward 1Gbit of UDP traffic in one direction only between two built in NIC's? Even though the packets are large, 1316 bytes payload, which gives a moderate 80..90kp/s in 1Gbit?

2

We abandoned the server, by spec the two on board network interfaces were not supposed to drive full gigabit traffic. The second interface was indented to be used for management.

A standard desktop core i5 with PCIe and two Intel i210 gigabit adapters was able to forward 1Gbit multicast UDP traffic with no problem.

Although, it required tweaking the RX and TX buffers (ethtool -G) due to burstiness in traffic. A 2x or 4x PCIe would probably help to reduce the risk of missed packets due to PCIe bus congestion.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.