4

We have a 3 Unix (1 Ubuntu 2 Redhat) servers connected via their 10 Gbit network cards to a 10 Gbit switch (our IT person made sure we got the correct cables). We experience some odd behaviour when sending files between the servers (either via scp 10Gbit-ip or a folder mounted via fstab using 10 Gbit ip). The speed is usually far from 10 Gbit but slightly over 1 Gbit. And on one of the server the progess bar is very quick but then gets stuck on 100 % for a long long time. Also true for scp on that server.

To make sure we got Full-Duplex and 10000 Mbit/s I used two different functions and they give conflicting information. Does someone have any ideas of what is going on? mii-tools and ethtool don't give the same information:

em1 = 1 Gbit network (via 10 Gbit port) em2 = 10 Gbit network (via 10 Gbit port)

ifconfig:

[admin@XXXX ~]$ ifconfig 
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXXX  netmask 255.255.255.0  broadcast XXXXX
        inet6 fe80::eef4:bbff:fed5:c230  prefixlen 64  scopeid 0x20<link>
        ether ec:f4:bb:d5:c2:30  txqueuelen 1000  (Ethernet)
        RX packets 3938043336  bytes 17103106879433 (15.5 TiB)
        RX errors 0  dropped 28  overruns 0  frame 0
        TX packets 8178327495  bytes 8360136815032 (7.6 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

em2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXXXX netmask 255.255.255.0  broadcast XXXX
        inet6 fe80::eef4:bbff:fed5:c232  prefixlen 64  scopeid 0x20<link>
        ether ec:f4:bb:d5:c2:32  txqueuelen 1000  (Ethernet)
        RX packets 3314001  bytes 28089131102 (26.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7632971  bytes 7936298958 (7.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 

mii-tool:

[admin@XXX ~]$ sudo mii-tool em1
[sudo] password for admin: 
SIOCGMIIREG on em1 failed: Invalid argument
SIOCGMIIREG on em1 failed: Invalid argument
SIOCGMIIREG on em1 failed: Invalid argument
em1: 1000 Mbit, half duplex, link ok

[admin@XXX ~]$ sudo mii-tool em2
SIOCGMIIREG on em2 failed: Invalid argument
SIOCGMIIREG on em2 failed: Invalid argument
SIOCGMIIREG on em2 failed: Invalid argument
em2: 1000 Mbit, half duplex, link ok

ethool:

[admin@XXXX ~]$ sudo ethtool em1
Settings for em1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

[admin@XXXX ~]$ sudo ethtool em2

Settings for em2:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

As you can see everything looks good with ethtool (full dubplex and full speed) but not with mii-tool (1000 Mb/s and half duplex). What could be the reason and potential test / fix?

  • 3
    mii-tool is outdated, ethtool is the way to go. As for (10Gbps) NICs' speed in general: there are some/many ways to enhance the speed. Jumbo frames, flow control, offloading, kernel tuning, etc. The used protocol/application affects the speed as well, of course. – Lenniey Jan 16 '18 at 13:03
  • 2
    Also don't test from disk-based-filesystem-to-disk-based-filesystem, test from a ramdisk of some form, something that's consistently faster than 10Gbps anyway - otherwise you're not testing your network on its own. – Chopper3 Jan 16 '18 at 13:08
  • @Lenniey Thanks! I'm trying to test the speed with iperf3 now but I'm having trouble for the servers to find each other. They can ping each other but gives: iperf3: error - unable to connect to server: No route to host when trying iperf3. I have asked our IT guy to check the firewall. Any pointers? – Robin Jan 16 '18 at 13:09
  • 1
    @Chopper3 Thanks! That is a really good tip. Trying to get iperf3 set-up so I can test the speed from there... Currently probable firewall issues. – Robin Jan 16 '18 at 13:10
  • Default iperf server settings use port 5001, so this port obviously shouldn't be blocked by your firewall. – Lenniey Jan 16 '18 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.