7

I have a Linux box that I use as a file server. I have a monthly cron job that tars up the contents of the data drive and then copies it via scp to another machine for safe keeping. The resulting tarball is around 300GB in size, and it normally takes about a day and a half to complete the copy (over an 802.11g Wi-Fi connection).

Today I noticed that my backup job hadn't completed yet, and it's been running for 3 days. Checking the destination machine, I could see that only about a third of the data has been copied so far, and it appears to be growing at a rate of < 300KB/sec.

Using iperf between the two machines, I can see that my network throughput is around 20Mbits/sec, which is about what I expect over an 802.11g connection.

Using dd if=srcfile of=/dev/null on the source machine, I can read about 45MB/sec from the source drive (an external USB drive).

using dd if=/dev/zero of=/destdrive/tmp.dat on the destination machine, I can write about 30MB/sec to the destination drive (internal SATA drive). Seems kind of slow for a SATA drive, but not unreasonably slow (and certainly not 300KB/sec slow).

So I seem to have ruled out network throughput and drive throughput at both ends, so where else can I look to find the source of the bottleneck?

Improve this question
4
  • It seems a network related problem. Try to transfer a test file (say 1GB) preferably with another protocol and measure transfer speed. – Vikelidis Kostas Nov 23 '16 at 13:43
  • I did that with iperf, which shows a 20Mbps throughput - about what you'd expect over an 802.11g link. – Jeff Loughlin Nov 23 '16 at 14:25
  • That is pretty slow for writes on the destination drive, even for a single sata spindle. Check the health of that disk using something like smartctl -a /dev/sd[X] to see if you have any pending or offline uncorrectable sectors, or if you have any read failures. – Spooler Nov 23 '16 at 14:51
  • 1
    @SmallLoanOf1M: This is a dinosaur machine with a SATA I interface, so the 30MB/sec write speed doesn't concern me too much. The kind of throughput I'm seeing is in the KB/sec range, so it doesn't seem like the drive is the bottleneck... that said, no errors on the drive that I can see. – Jeff Loughlin Nov 23 '16 at 17:51
9

Why you are using scp for copying large files in the first place? scp has its own overhead (encryption, authenticity checking, etc).

You can use rsync (rsync is very well suited for transferring large files over ssh because it is able to continue transfers that were interrupted due to some reason. Since it uses hash functions to detect equal file blocks the continue feature is quite robust.) or some other tool.

Please see this post. Copying large files over network , faster

If you want to use scp anyway , then you should use traceroute and tcpdump and iftopto see the packets from source to destination. May be you find something unusual.

Improve this answer
9
  • Fair enough. But this backup script has been running for years using scp, and completing in a day and a half (which is perfectly acceptable to me, as it's just an unattended backup job that runs in the background once a month). I'm looking for tools to diagnose what the bottleneck is, because something seems to have changed. – Jeff Loughlin Nov 23 '16 at 14:28
  • rsync is your answer if you ask me. – Itai Ganot Nov 23 '16 at 14:33
  • can you traceroute from sender to reciever and confirm that the path is what is expected to be – Ijaz Ahmad Khan Nov 23 '16 at 14:35
  • Plus , can you try scp with compression enabled. see man scp – Ijaz Ahmad Khan Nov 23 '16 at 14:38
  • answer updated. – Ijaz Ahmad Khan Nov 23 '16 at 14:47
2

Check to make sure -l option is not enabled to limit the bandwidth. Also, it looks like there is a -v will give insight to what is going on for the next run.

Verbose mode. Causes scp and ssh(1) to print debugging messages about their progress. This is helpful in debugging connection, authentication, and configuration problems.

This has been answered before. Quote from the answer.

scp is using interactive terminal in order to print that fancy progress bar. Printing that output to a file does not make sense at all, so scp detects when its output is redirected to somewhere else other than a terminal and does disable this output.

Full answer

https://stackoverflow.com/questions/3890809/bash-stdout-redirect-of-commands-like-scp

SCP man page

https://linux.die.net/man/1/scp

Improve this answer
0

I also encountered slow SCP performance while coping files ~150-300KiB/s instead of 10MiB/s. Also I noticed that on target server 1 CPU core was busy 100% while I was coping a file. I googled a bit and found proposal: disable "Optimize connection buffer size" in SCP connection options. It helped. After disabling this option speed increased to expected network level, CPU load on server significatly reduced.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.