5

I've got two linux boxes attached to a gigabit switch. They both have gigabit NICs, cables are cat7.

Testing the network with iperf shows a fast connection but transferring files with rsync, scp, or nfs share is slow.

I'm testing with one 1GB file.

iperf result:

Client connecting to odroid, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.26 port 58788 connected with 192.168.1.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   979 MBytes   821 Mbits/sec

The transfer speed with rsync, scp or nfs is all about ~13Mb/s

scp:

 scp bigfile odroid:/mnt/usb1/               [10:19:12]
bigfile                                        57%  590MB  12.2MB/s   00:35 ETA^CKilled by signal 2.

rsync:

 rsync --progress bigfile /mnt/usb1/
bigfile
     44,695,552   4%   12.15MB/s    0:01:11  ^C

nfs:

binaryplease➜~(master✗)» time cp bigfile /mnt/nfs/usb1/      
cp -i bigfile /mnt/nfs/usb1/  0.01s user 0.94s system 1% cpu 1:11.06 total

1024MB/71sec = 14,42 MB/s

Since the iperf test shows a fast network connection, I assumed a problem with the storage devices being slow, but that doesn't seem to be the case either:

Client, SSD, internal:

binaryplease➜~(master✗)» sudo hdparm -tT /dev/sda                    
/dev/sda:
 Timing cached reads:   20344 MB in  2.00 seconds = 10181.50 MB/sec
 Timing buffered disk reads: 1498 MB in  3.00 seconds = 498.98 MB/sec

binaryplease➜~(master✗)» dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 2.03861 s, 263 MB/s

binaryplease➜~(master✗)» dd if=test of=/dev/null iflag=direct bs=8M  [12:29:01]
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 1.11392 s, 482 MB/s

Server, USB 3.0 Drive, external:

➜  ~ git:(master) ✗ sudo hdparm -tT /dev/sda   
/dev/sda:
 Timing cached reads:   1980 MB in  2.00 seconds = 991.66 MB/sec
 Timing buffered disk reads: 266 MB in  3.01 seconds =  88.27 MB/sec
➜  usb1   dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 6.53386 s, 82.2 MB/s
➜  usb1  dd if=test of=/dev/null iflag=direct bs=8M
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 7.13567 s, 75.2 MB/s

OS on client (Linux arch):

Linux binaryplease-laptop 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015 x86_64 GNU/Linux

OS on server (Ubuntu server for odroid):

Linux odroid 3.10.92 #1 SMP PREEMPT Tue Nov 17 00:15:24 BRST 2015 armv7l armv7l armv7l GNU/Linux

On both systems neither the cpu or the ram is maxed out.

If I interpret the results correctly the write speed of the servers drive (82.2 MB/s) should be easily matched by the network. How is the file transfer so slow?

I hope the information provided is sufficient and someone can help me find the bottleneck.

Thanks.

0

At the numbers you get, seems like something is not actually gigabit in your network.

Use ethtool or something similar to check the speed on each machine NIC.

Check the switch and make sure the ports are actually active at gigabit speed (check LEDs or configuration port status if the switch supports management).

Run iperf or an alternate between the clients.

Finally, check CPU usage - perhaps something is wrong there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.