4

I want to test what's the max netwotk bandwidth our Linux server can handle. I use one Dell R710 with two quad-core Intel cpu and 16GB memory. I install two Intel 82575Gb pcie cards, each has 4 ports And onboard there are 4 Broadcom BCM5709Gb ports. So I have total 12 Gb nic ports.

I wrote a simple udp test program to send non-stop dummy udp data through specific nic using connect(). So when I start one program to send udp data through one nic, I can see network throughput is around 116MB/s for that nic. This is resonable result. Then I start another test program to send udp using another nic.

The final max network throughput I got is around 1GB/s. That means I can start only 10 test programs. One more started program will downgrade throughput dramatically.

My question is, is that possible to use all 16 ports to send data in full speed? Is there any throughput limitation Linux supports?

I modify some udp sysctl parameter but didn't work. The memory seems big enough, and CPU still has enough resource. Can anyone helps me how to tweak system?

How about 10Gb nic? If I install two of them I can't get them reach full speed.

Any help is welcome.

Thanks

2 Answers 2

3

Just on the topic of maxing traffic, instead of writing your own udp test program, iperf (iperf package in Debian/Ubuntu) is a great tool to generate TCP bandwidth, but it require a remote host with Linux with same or better network capacity:

Remote host:

# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------

On host I want to test:

iperf -i 1 -c 192.168.15.6
------------------------------------------------------------
Client connecting to 192.168.15.6, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.15.132 port 37163 connected with 192.168.15.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  11.4 MBytes  95.2 Mbits/sec
[  3]  1.0- 2.0 sec  11.4 MBytes  95.3 Mbits/sec
[  3]  2.0- 3.0 sec  11.1 MBytes  93.1 Mbits/sec
[  3]  3.0- 4.0 sec  11.3 MBytes  94.8 Mbits/sec
[  3]  4.0- 5.0 sec  11.2 MBytes  94.2 Mbits/sec
[  3]  5.0- 6.0 sec  11.1 MBytes  93.5 Mbits/sec
[  3]  6.0- 7.0 sec  11.3 MBytes  95.0 Mbits/sec
[  3]  7.0- 8.0 sec  11.4 MBytes  95.3 Mbits/sec
[  3]  8.0- 9.0 sec  11.4 MBytes  95.2 Mbits/sec
[  3]  9.0-10.0 sec  11.7 MBytes  97.8 Mbits/sec
[  3]  0.0-10.0 sec    113 MBytes  94.6 Mbits/sec
2

It looks like you're hitting the maximum bus bandwidth. The server has two PCIe x8 slots, each of which will just about let you max out 4 GB ports. The onboard Broadcom ports are probably connected with plain old PCI, so you're not going to be able to max those out - not even close.

You're right that it won't be able to max out two 10GB/s NICs. The hardware just isn't capable.

You must log in to answer this question.