4

I got Intel X520 and connected it to PCI-E x16. The card is recognized as 10 Gbit by ethtool: 

 # ethtool eth5
Settings for eth5:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full 
                                10000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  10000baseT/Full 
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

The switch also recognizes the link as 10 Gbit. I ran iperf tests from 12 different machines (2 times x 6). Every single connection is 1 Gbit, except the server one. So, I expect to see around 6 Gbit total (6 machines x 1 Gbit each). I got only total of 1 Gbit on the server side. My questions are:

  1. am I doing something wrong?
  2. how to properly test the new card?

The OS is Ubuntu Karmic, the MB is Supermicro X8DAi. Thanks!

EDIT: The switch is single Netgear GSM7328S-200NAS. Here is a diagram: 6 1 Gbps Clients to 10 Gbps server

I test by running:

Server: iperf -s
All clients, simultaneously: iperf -c server_IP

I see values around 150-160 Mbps/client, which match to 1 Gbps link to the server. The IP I assigned to the new card was never used, so I suppose no ARP issues. However, all traffic is on one subnet, as the same clients use the old 1 Gbps card and IP of the server.

No virtualization here, just plain Ubuntu install, serving NFS mostly.

Update: after further tests, I found that the clients' ARP tables contain wrong entry for server's 10 Gbps NIC. The HW address listed there is the same as the server's 1 Gbps NIC. I deleted it and manually inserted the correct entry: arp -d server_IP_10g, arp -s server_IP_10g HW:address. After that I was unable to ping server_IP_10g.

How can I fix it without disabling the 1 Gbps NIC in the server?

UPDATE2: sysctl -w net.ipv4.conf.all.arp_ignore=1 sysctl -w net.ipv4.conf.all.arp_announce=2 does not work.

Improve this question
7
  • Can you provide more details about the switching hardware involved and how the servers are interconnected on that switching hardware(which blades, trunks, supervisors, etc)?
    – polynomial
    Sep 8, 2011 at 23:50
  • Could you also describe your test methodology? It's not quite clear.
    – sybreon
    Sep 9, 2011 at 0:45
  • @polynomial - I updated my question to provide the requested info. Thanks!
    – grs
    Sep 9, 2011 at 2:02
  • @sybreon - I updated my question to provide the requested info. Thanks!
    – grs
    Sep 9, 2011 at 2:02
  • 2
    You may find my question at serverfault.com/questions/254773/… helpful.
    – sciurus
    Sep 9, 2011 at 18:51

1 Answer 1

Reset to default
2

After carefully revising pros and cons and what actually is possible to test, I went and disabled eth3 on the server (the 1 Gbps NIC) and set up eth5 (the 10 Gbps) with its address. Then the clients found the new interface and iperf showed 10 Gbps from 10 x 1 Gbps clients. So, all is good. I think @sciurus' question, posted above, would be great, if I needed both interfaces permanently. +1 from me on the good reference!

Improve this answer
1
  • +1 for coming back and answering your question for future people to find!
    – Mark Henderson
    Sep 13, 2011 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.