4

I got Intel X520 and connected it to PCI-E x16. The card is recognized as 10 Gbit by ethtool: 

 # ethtool eth5
Settings for eth5:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full 
                                10000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  10000baseT/Full 
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

The switch also recognizes the link as 10 Gbit. I ran iperf tests from 12 different machines (2 times x 6). Every single connection is 1 Gbit, except the server one. So, I expect to see around 6 Gbit total (6 machines x 1 Gbit each). I got only total of 1 Gbit on the server side. My questions are:

  1. am I doing something wrong?
  2. how to properly test the new card?

The OS is Ubuntu Karmic, the MB is Supermicro X8DAi. Thanks!

EDIT: The switch is single Netgear GSM7328S-200NAS. Here is a diagram: 6 1 Gbps Clients to 10 Gbps server

I test by running:

Server: iperf -s
All clients, simultaneously: iperf -c server_IP

I see values around 150-160 Mbps/client, which match to 1 Gbps link to the server. The IP I assigned to the new card was never used, so I suppose no ARP issues. However, all traffic is on one subnet, as the same clients use the old 1 Gbps card and IP of the server.

No virtualization here, just plain Ubuntu install, serving NFS mostly.

Update: after further tests, I found that the clients' ARP tables contain wrong entry for server's 10 Gbps NIC. The HW address listed there is the same as the server's 1 Gbps NIC. I deleted it and manually inserted the correct entry: arp -d server_IP_10g, arp -s server_IP_10g HW:address. After that I was unable to ping server_IP_10g.

How can I fix it without disabling the 1 Gbps NIC in the server?

UPDATE2: sysctl -w net.ipv4.conf.all.arp_ignore=1 sysctl -w net.ipv4.conf.all.arp_announce=2 does not work.

Improve this question
7
  • Can you provide more details about the switching hardware involved and how the servers are interconnected on that switching hardware(which blades, trunks, supervisors, etc)?
    – polynomial
    Sep 8, 2011 at 23:50
  • Could you also describe your test methodology? It's not quite clear.
    – sybreon
    Sep 9, 2011 at 0:45
  • @polynomial - I updated my question to provide the requested info. Thanks!
    – grs
    Sep 9, 2011 at 2:02
  • @sybreon - I updated my question to provide the requested info. Thanks!
    – grs
    Sep 9, 2011 at 2:02
  • 2
    You may find my question at serverfault.com/questions/254773/… helpful.
    – sciurus
    Sep 9, 2011 at 18:51

1 Answer 1

Reset to default
2

After carefully revising pros and cons and what actually is possible to test, I went and disabled eth3 on the server (the 1 Gbps NIC) and set up eth5 (the 10 Gbps) with its address. Then the clients found the new interface and iperf showed 10 Gbps from 10 x 1 Gbps clients. So, all is good. I think @sciurus' question, posted above, would be great, if I needed both interfaces permanently. +1 from me on the good reference!

Improve this answer
1
  • 1
    +1 for coming back and answering your question for future people to find!
    – Mark Henderson
    Sep 13, 2011 at 2:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .