I got Intel X520 and connected it to PCI-E x16. The card is recognized as 10 Gbit by ethtool:
# ethtool eth5 Settings for eth5: Supported ports: [ FIBRE ] Supported link modes: 1000baseT/Full 10000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10000baseT/Full Advertised auto-negotiation: Yes Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 0 Transceiver: external Auto-negotiation: on Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) Link detected: yes
The switch also recognizes the link as 10 Gbit.
iperf tests from 12 different machines (2 times x 6). Every single connection is 1 Gbit, except the server one. So, I expect to see around 6 Gbit total (6 machines x 1 Gbit each). I got only total of 1 Gbit on the server side.
My questions are:
- am I doing something wrong?
- how to properly test the new card?
The OS is Ubuntu Karmic, the MB is Supermicro X8DAi. Thanks!
EDIT: The switch is single Netgear GSM7328S-200NAS. Here is a diagram:
I test by running:
Server: iperf -s All clients, simultaneously: iperf -c server_IP
I see values around 150-160 Mbps/client, which match to 1 Gbps link to the server. The IP I assigned to the new card was never used, so I suppose no ARP issues. However, all traffic is on one subnet, as the same clients use the old 1 Gbps card and IP of the server.
No virtualization here, just plain Ubuntu install, serving NFS mostly.
Update: after further tests, I found that the clients' ARP tables contain wrong entry for server's 10 Gbps NIC. The HW address listed there is the same as the server's 1 Gbps NIC. I deleted it and manually inserted the correct entry:
arp -d server_IP_10g,
arp -s server_IP_10g HW:address. After that I was unable to
How can I fix it without disabling the 1 Gbps NIC in the server?
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2 does not work.