We use rsync as backup on a NFS share we have on our VPN network.
The ISP's modem is awful and it stops responding when we ask too much upload bandwidth to it. We have 150Mbps down / 20 Mbps up according to speedtest.
The number of files and size varies greatly, rsync checks 200.000 files, in one of the backups, of very variate sizes, from KB to 50 GB or more.
To avoid this, we use rsync's bwlimit option. The problem is that apparently even 10s bursts can make the modem irresponsive.
From its manual entry:
Due to the nature of rsync transfers, blocks of data are sent, then if rsync determines the transfer was too fast, it will wait before sending the next data block. The result is an average transfer rate equaling the specified limit.
Example of crontab line for the backup of /var/www
flock rsync_wan_lock -c "nice -n1 ionice -c2 -n2 rsync -aq --partial --bwlimit 1000K --delete-after /var/www /mnt/foo_bk/."
Is there any way to mitigate or solve this?
rsyncoptions you use? What is the available upload bandwidth? What bwlimit value did you try? How many files are you uploading? What is the total size of the files?flock(that's the reason behind it). Lowering thebwlimitmakes the problem come less frequently, but it's hard to tell, I'm unsure how the size of the blocks written is decided by rsync.