Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running some command in cron that does an rsync over two data centers. The connection rate between the data centers is around 1Mbps. I noticed that rsync keeps failing around 2.7GB to 2.9GB of transferred data, which could equal around an hour or more on said connection. I know in the past that our firewall is configured to close idle connections that have opened for more than an hour.

But would an rsync transfer really be idle? What can I do to fix this?

Error:

rsync: connection unexpectedly closed (2987596424 bytes received so far) [receiver]

rsync error: error in rsync protocol data stream (code 12) at io.c(605) [receiver=3.0.9]

rsync: connection unexpectedly closed (523 bytes received so far) [generator]

rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9]

I already have this in my ~/.ssh/config (but maybe these values need to be increased?):

Host *
  ServerAliveInterval 60
  ServerAliveCountMax 15
share|improve this question
    
Don't get the downvote, but OK. –  Bradford Jul 28 '13 at 13:49
    
Do you know what your firewall's exact idle rule is? If you do, you can configure ssh to send keep alives often enough not to hit it. –  David Schwartz Jul 28 '13 at 18:39
    
I believe this was caused by network saturation. –  Bradford Jul 31 '13 at 13:47

1 Answer 1

up vote 4 down vote accepted

Idle network connection during rsync transfert:

Yes, rsync could stay quiet during disk read and while they have no packet to send...

You said. our firewall is configured to close idle connections that have opened for more than an hour but nothing about the continuous idle duration.

rsync process on both ends, reading whole files to search for difference between packets, based on checksums previously computed.

While there are no difference between files on both ends, they may be no exchange between both ends until differences are found.

There is a little graphic representing a single rsync run between my desktop and his backup:

Trace rsync network traffic

In which we may see some seconds elapsed without datas:

TIME                     BLOCKS           BYTES
                        IN  OUT      IN     OUT
...
Jul 28 15:00:07 2013   449   41   23443 1526403
Jul 28 15:00:08 2013     0    0       0       0
Jul 28 15:00:09 2013     0    0       0       0
Jul 28 15:00:10 2013   143  182   21348 1123986
Jul 28 15:00:11 2013   112  142    6966  523326
Jul 28 15:00:12 2013    31   30    1942    1890
Jul 28 15:00:13 2013    30   28    1853   88038
Jul 28 15:00:14 2013     0    0       0       0
Jul 28 15:00:15 2013    57   66    3954   38301
...

Workaround:

As a workaround, you may use socket based ssh sharing:

First run a plain ssh for hanging socket and connection

ssh -M -o ControlPath=$HOME/.ssh/socket_test destHost ping -n localhost

Than in another window:

rsync -e 'ssh -e none -S $HOME/.ssh/socket_test' /src/file destHost:/dst/file

In this way, only one connection will be used for both ssh sessions. The ping will ensure (a lot of) continuous traffic, than rsync could work quietely, using same network connection.

share|improve this answer
    
Thanks a lot. I'm using some replication tool that uses rsync and cannot easily use the workaround. This was a good explanation of what's going on, though. –  Bradford Jul 28 '13 at 17:15
    
By default, ssh don't run as ControMaster, so you may just have to run ssh -M destHost ping -n localhost before running rsync in normal way. Not sure, but this may work! –  F. Hauri Jul 28 '13 at 17:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.