I'm using rsync to perform backups from an NFS server hosted in AWS to another EC2 instance in the same AZ, the rsync command that I'm using in the backup server is the following:

 rsync -avzb --backup-dir=someDirectory* /opt/destination/

My backup consists of several thousands, or may be more than a millon, small files (.pdfs around 200/500 KB)

The issue that I'm having is that the incremental file list is sent very quickly (so far so good) but when rsync starts copying files is painfully slow, it copies like or 20 files, then it stops for 3 or 4 minutes, then it copies some more files, and so on.

I'm running the rsync process in a crontab every 2 hours and most of the times I have a very long list of unfinished rsync processes which forces me to reboot the server.

This is my iowait:

Linux 4.4.0-1060-aws (prd-turecibo-backup)  06/06/2018  _x86_64_    (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.29    0.00    0.74   84.56    0.00   14.41

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
nvme1n1           0.22         6.00         0.00      20984          0
nvme2n1         240.45       959.12        17.95    3354430      62788
nvme0n1           3.43        65.19         2.03     227995       7112

Usually iowait is at 90%.

I've replaced the EBS volume and the problem persists, algo upgraded the intance but no luck.

Any ideas?

  • 1
    What volume type and how large are you using for EBS? – B. Miller Jun 7 '18 at 3:12
  • You're going to need a bigger boa^WEBS IOPS allocation. – womble Jun 7 '18 at 4:31
  • iostat -x 1 will provide more meaningful numbers, including the approximate % utilization of each device. – Michael - sqlbot Jun 7 '18 at 13:18
  • 1
    Which filesystem is NFS, local or remote? With rsync it's recommended to not read/write NFS filesystems but preferably run rsync against the NFS server's local storage, as rsync is designed to optimize for network bandwith, sacrificing disk IO; however NFS IO is network IO as well, so it is very suboptimal. – wurtel Jun 7 '18 at 14:07
  • if you are using amazon efs, check the volume burst – c4f4t0r Jun 7 '18 at 18:39

I'm running the rsync process in a crontab every 2 hours and most of the times I have a very long list of unfinished rsync processes which forces me to reboot the server.

Start with making only one instance of rsync-backup running
e.g. by using flock command in shell script.

You may alternatively use run-one if you distribution provides it. [suggested by Michael - sqlbot]
It seems to be provided by Linux/Ubuntu.

  • Or x x x x x run-one rsync ... in the crontab, which provides the same thing, without separate a wrapper script. It's possible that if many rsyncs are running, that's exactly the root cause... they are all essentially wasting iops. – Michael - sqlbot Jun 7 '18 at 13:21
  • Thanks for the suggestion, I'll try that and let you know. – nikolaigauss Jun 7 '18 at 20:57
  • That did the trick! I managed to ask the PID using 'pidof' to prevent 2 instances of the same rsync execution to run in parallel. Thanks! – nikolaigauss Jun 18 '18 at 2:03
  • Have it helped to solve (more than reduce) main problem? [very slow rsync] – AnFi Jun 18 '18 at 7:02

The other thing to consider is disabling compression. We were attempting to transfer a DB (MySQL/MariaDB) between 2 ec2 nodes in AWS and were seeing this transfer run around 15-20MB/s.

NOTE: We were expecting to see an order of magnitude higher with the m4.2xlarge instances we were using, they were in 2 different AZs but given all this smaller transfers such as these were showing no limiting being imposed by AWS:

$ truncate -s 500MB 500MB.file
$ ll | grep 500MB
-rw-rw-r--. 1 smingolelli smingolelli 500000000 Sep  5 14:55 500MB.file

$ rsync --progress -h -v 500MB.file
        500.00M 100%   81.55MB/s    0:00:05 (xfr#1, to-chk=0/1)

sent 500.12M bytes  received 35 bytes  66.68M bytes/sec
total size is 500.00M  speedup is 1.00

We repeated the test with compression off and saw the transfer numbers climb to what we'd expect. So you might want to try your command without the -z:

$ rsync -avb --backup-dir=someDirectory* /opt/destination/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.