rsync connection closing right around an hour

Question

I am running some command in cron that does an rsync over two data centers. The connection rate between the data centers is around 1Mbps. I noticed that rsync keeps failing around 2.7GB to 2.9GB of transferred data, which could equal around an hour or more on said connection. I know in the past that our firewall is configured to close idle connections that have opened for more than an hour.

But would an rsync transfer really be idle? What can I do to fix this?

Error:

rsync: connection unexpectedly closed (2987596424 bytes received so far) [receiver]

rsync error: error in rsync protocol data stream (code 12) at io.c(605) [receiver=3.0.9]

rsync: connection unexpectedly closed (523 bytes received so far) [generator]

rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9]

I already have this in my ~/.ssh/config (but maybe these values need to be increased?):

Host *
  ServerAliveInterval 60
  ServerAliveCountMax 15

Answer 1

Idle network connection during rsync transfert:

Yes, rsync could stay quiet during disk read and while they have no packet to send...

You said. our firewall is configured to close idle connections that have opened for more than an hour but nothing about the continuous idle duration.

rsync process on both ends, reading whole files to search for difference between packets, based on checksums previously computed.

While there are no difference between files on both ends, they may be no exchange between both ends until differences are found.

There is a little graphic representing a single rsync run between my desktop and his backup:

In which we may see some seconds elapsed without datas:

TIME                     BLOCKS           BYTES
                        IN  OUT      IN     OUT
...
Jul 28 15:00:07 2013   449   41   23443 1526403
Jul 28 15:00:08 2013     0    0       0       0
Jul 28 15:00:09 2013     0    0       0       0
Jul 28 15:00:10 2013   143  182   21348 1123986
Jul 28 15:00:11 2013   112  142    6966  523326
Jul 28 15:00:12 2013    31   30    1942    1890
Jul 28 15:00:13 2013    30   28    1853   88038
Jul 28 15:00:14 2013     0    0       0       0
Jul 28 15:00:15 2013    57   66    3954   38301
...

Workaround:

As a workaround, you may use socket based ssh sharing:

First run a plain ssh for hanging socket and connection

ssh -M -o ControlPath=$HOME/.ssh/socket_test destHost ping -n localhost

Than in another window:

rsync -e 'ssh -e none -S $HOME/.ssh/socket_test' /src/file destHost:/dst/file

In this way, only one connection will be used for both ssh sessions. The ping will ensure (a lot of) continuous traffic, than rsync could work quietely, using same network connection.