I've been scratching my head over this for the past 1 week and can't seem to find any solution. So, this my problem... I have a Juniper ex4500 10g switch connected to my servers over sfp+ twinax cables.
1st server (filer): Dell R620
RAM : 96GB N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) on bond0 which makes this 20Gbps OS : centos 6.7 x86_64 Storage : 24 disks connected via perc 810 on RAID 5
2st server (compute): Dell R810
RAM : 256GB N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) no bond OS : centos 6.7 x86_64 Storage : single disk 512GB 6Bb/s
3rd server (compute): Dell R810
RAM : 512GB N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) no bond OS : centos 6.7 x86_64 Storage : single disk 512GB 6Bb/s
I have a CentOS 6.7 ISO as the file that I'm using for testing the transfer speed. And this is how I started my tests. First I started with server 2 and server 3 by using
scp centos 6.7.iso root@server3:~/ and the speed it is transferring is maxing out to 167MB/sec. I figured it was because the MTU is 1500 and I'm getting lower speeds. So changed the MTU on the switch to 9214 on the ports the servers are connected to and in the servers ifcfg-px files and rebooted the servers. But now the transfer speed increased only to 200MB/sec. I tested with server 1 which has a bond on it and MTU set to 9000 I still get 200MB/s.
iperf says 10g as the bandwidth but I can't see it. Does scp have so much of an overhead that it is maxing out to 200MB/s?
Anyways, if anyone of you have any idea what the problem is please let me know. it is highly appreciated.
Below are my outputs of iperf and scp.
Client connecting to teng.tofu, TCP port 5001 TCP window size: 92.6 KByte (default) [ 3] local 10.10.113.15 port 58221 connected with 10.10.113.21 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 11.5 GBytes 9.90 Gbits/sec
100% 3926MB 186.9MB/s 00:21