Debug slow lan (ssh, nfs) file transfer

Question

I've got two linux boxes attached to a gigabit switch. They both have gigabit NICs, cables are cat7.

Testing the network with iperf shows a fast connection but transferring files with rsync, scp, or nfs share is slow.

I'm testing with one 1GB file.

iperf result:

Client connecting to odroid, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.26 port 58788 connected with 192.168.1.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   979 MBytes   821 Mbits/sec

The transfer speed with rsync, scp or nfs is all about ~13Mb/s

scp:

 scp bigfile odroid:/mnt/usb1/               [10:19:12]
bigfile                                        57%  590MB  12.2MB/s   00:35 ETA^CKilled by signal 2.

rsync:

 rsync --progress bigfile /mnt/usb1/
bigfile
     44,695,552   4%   12.15MB/s    0:01:11  ^C

nfs:

binaryplease➜~(master✗)» time cp bigfile /mnt/nfs/usb1/      
cp -i bigfile /mnt/nfs/usb1/  0.01s user 0.94s system 1% cpu 1:11.06 total

1024MB/71sec = 14,42 MB/s

Since the iperf test shows a fast network connection, I assumed a problem with the storage devices being slow, but that doesn't seem to be the case either:

Client, SSD, internal:

binaryplease➜~(master✗)» sudo hdparm -tT /dev/sda                    
/dev/sda:
 Timing cached reads:   20344 MB in  2.00 seconds = 10181.50 MB/sec
 Timing buffered disk reads: 1498 MB in  3.00 seconds = 498.98 MB/sec

binaryplease➜~(master✗)» dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 2.03861 s, 263 MB/s

binaryplease➜~(master✗)» dd if=test of=/dev/null iflag=direct bs=8M  [12:29:01]
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 1.11392 s, 482 MB/s

Server, USB 3.0 Drive, external:

➜  ~ git:(master) ✗ sudo hdparm -tT /dev/sda   
/dev/sda:
 Timing cached reads:   1980 MB in  2.00 seconds = 991.66 MB/sec
 Timing buffered disk reads: 266 MB in  3.01 seconds =  88.27 MB/sec
➜  usb1   dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 6.53386 s, 82.2 MB/s
➜  usb1  dd if=test of=/dev/null iflag=direct bs=8M
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 7.13567 s, 75.2 MB/s

OS on client (Linux arch):

Linux binaryplease-laptop 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015 x86_64 GNU/Linux

OS on server (Ubuntu server for odroid):

Linux odroid 3.10.92 #1 SMP PREEMPT Tue Nov 17 00:15:24 BRST 2015 armv7l armv7l armv7l GNU/Linux

On both systems neither the cpu or the ram is maxed out.

If I interpret the results correctly the write speed of the servers drive (82.2 MB/s) should be easily matched by the network. How is the file transfer so slow?

I hope the information provided is sufficient and someone can help me find the bottleneck.

Thanks.

Answer 1

At the numbers you get, seems like something is not actually gigabit in your network.

Use ethtool or something similar to check the speed on each machine NIC.

Check the switch and make sure the ports are actually active at gigabit speed (check LEDs or configuration port status if the switch supports management).

Run iperf or an alternate between the clients.

Finally, check CPU usage - perhaps something is wrong there.

Answer 2

Thanks so much for this post, I was experiencing exactly the same issue... So, I'd just like to expand on:

seems like something is not actually gigabit in your network.

So, it turns out, it was my Wi-Fi adapter on my laptop, that may have been the issue.

First I looked for a way to check speed of network adapters, and consulted https://www.systutorials.com/how-to-find-a-wireless-network-adapters-speed-in-linux/ - it suggests ethtool, but that does not work for wireless network adapters.

Then I found:

• https://askubuntu.com/questions/733276/wireless-adapter-speed-changes

So, I had:

$ iwconfig wlp3s0 | grep Rate
          Bit Rate=270 Mb/s   Tx-Power=15 dBm  

$ rsync -aP /nfsshare/test.zip .
sending incremental file list
test.zip
     51,412,992  17%   12.16MB/s    0:00:19  ^C

Then I tried:

$ sudo iw dev wlp3s0 get power_save
Power save: on

$ sudo iw dev wlp3s0 set power_save off
$ sudo iw dev wlp3s0 get power_save
Power save: off

...a then, after a couple of rsyncs, I got this:

$ rsync -aP /nfsshare/test.zip .
sending incremental file list
test.zip
    292,988,598 100%  259.41MB/s    0:00:01 (xfr#1, to-chk=0/1)

Not sure if something got cached along the way, but I got impressed by those numbers.

Then again, git status on the same PC/laptop, for a git project in the same NFS share, still seems as slow as ever ...