3

I have an issue which I am struggling to understand. I have a server which is hosted by a business parter on their network but my team is responsible for installing and configuring software to this server (actually there are 3 servers all exhibiting the same behaviour). As part of the configuration we are trying to copy files via rsync (over SSH) but we are running into issues which we and out partner do not understand.

Essentially it seems we are able to rsync files that are less than 32768 bytes but after that the connection stalls. We are doing this over SSH and I can get a responsive shell on the server. I am connecting across the internet but I have tried from two locations with the same results. Here is an example of what I see:

rsync -aP ~/Downloads/file.zip servername:~ -vvv
opening connection using ssh servername rsync --server -vvvlogDtpr --partial . "~"
building file list ...
[sender] make_file(file.zip,*,2)
1 file to consider
server_recv(2) starting pid=2610
send_file_list done
send_files starting
received 1 names
recv_file_list done
get_local_name count=1 /home/me
generator starting pid=2610
delta-transmission enabled
recv_files(1) starting
recv_generator(file.zip,0)
send_files(0, /Users/me/Downloads/file.zip)
send_files mapped /Users/me/Downloads/file.zip of size 54965002
calling match_sums /Users/me/Downloads/file.zip
file.zip
       32768   0%    0.00kB/s    0:00:00

This will stall for some minutes and eventually timeout. I have captured the packets from my side while I'm not particularly versed in reading packet captures it seems that the server side stops responding for a few minutes and eventually resets the connection. I found a tshark snippet on another question which I tweaked slightly to get this:

tshark -r ~/rsync-timeout.pcap -q -z io,stat,20,"COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission","COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack","COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment","COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission"

===================================================================================
| IO Statistics                                                                   |
|                                                                                 |
| Duration: 395.924510 secs                                                       |
| Interval:  20 secs                                                              |
|                                                                                 |
| Col 1: COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission           |
|     2: COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack              |
|     3: COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment               |
|     4: COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission |
|---------------------------------------------------------------------------------|
|            |1      |2      |3      |4      |                                    |
| Interval   | COUNT | COUNT | COUNT | COUNT |                                    |
|--------------------------------------------|                                    |
|   0 <>  20 |     0 |     0 |     0 |     0 |                                    |
|  20 <>  40 |     2 |     1 |     0 |     0 |                                    |
|  40 <>  60 |    13 |     0 |     0 |     0 |                                    |
|  60 <>  80 |     4 |     0 |     0 |     0 |                                    |
|  80 <> 100 |     4 |     0 |     0 |     0 |                                    |
| 100 <> 120 |     0 |     0 |     0 |     0 |                                    |
| 120 <> 140 |     4 |     0 |     0 |     0 |                                    |
| 140 <> 160 |     0 |     0 |     0 |     0 |                                    |
| 160 <> 180 |     0 |     0 |     0 |     0 |                                    |
| 180 <> 200 |     4 |     0 |     0 |     0 |                                    |
| 200 <> 220 |     0 |     0 |     0 |     0 |                                    |
| 220 <> 240 |     4 |     0 |     0 |     0 |                                    |
| 240 <> 260 |     0 |     0 |     0 |     0 |                                    |
| 260 <> 280 |     4 |     0 |     0 |     0 |                                    |
| 280 <> 300 |     0 |     0 |     0 |     0 |                                    |
| 300 <> 320 |     4 |     0 |     0 |     0 |                                    |
| 320 <> 340 |     0 |     0 |     0 |     0 |                                    |
| 340 <> 360 |     4 |     0 |     0 |     0 |                                    |
| 360 <> 380 |     0 |     0 |     0 |     0 |                                    |
| 380 <> Dur |     0 |     0 |     0 |     0 |                                    |
===================================================================================

I can see that that isn't great but I'm not really sure what it tells me. I can see in the packet trace that there is no response from the server (or none getting to me) for a few minutes and then finally a RST is set and both sides hang up.

The end of my packet trace as viewed with tshark looks like this:

... everything seems ok before this point

429  45.647732 1.2.3.4 -> 192.168.1.18 TCP 66 22→53839 [ACK] Seq=2438 Ack=15846 Win=60288 Len=0 TSval=8701862 TSecr=552453169
430  45.714775 1.2.3.4 -> 192.168.1.18 TCP 66 22→53839 [ACK] Seq=2438 Ack=17254 Win=63232 Len=0 TSval=8701928 TSecr=552453237
431  45.748600 1.2.3.4 -> 192.168.1.18 TCP 66 22→53839 [ACK] Seq=2438 Ack=18662 Win=64128 Len=0 TSval=8701963 TSecr=552453237
432  45.821013 1.2.3.4 -> 192.168.1.18 TCP 66 22→53839 [ACK] Seq=2438 Ack=21478 Win=64128 Len=0 TSval=8702035 TSecr=552453237
433  47.331298 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
434  49.243984 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
435  49.243989 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
436  49.244199 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
437  49.244203 192.168.1.18 -> 1.2.3.4 SSHv2 882 Client: [TCP Retransmission] , Encrypted packet (len=816)
438  52.678673 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
439  52.678677 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)

... more of the same ...

472 309.358046 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
473 309.358166 192.168.1.18 -> 1.2.3.4 SSHv2 222 Client: [TCP Retransmission] , Encrypted packet (len=156)
474 335.714554 1.2.3.4 -> 192.168.1.18 TCP 60 22→53837 [RST, ACK] Seq=4050 Ack=5018 Win=0 Len=0
475 352.579591 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
476 352.579592 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
477 352.579595 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
478 352.579596 192.168.1.18 -> 1.2.3.4 SSHv2 222 Client: [TCP Retransmission] , Encrypted packet (len=156)
479 395.924510 192.168.1.18 -> 1.2.3.4 TCP 54 53839→22 [RST, ACK] Seq=29014 Ack=2438 Win=131072 Len=0

I would love to have some ideas on what we can do to troubleshoot this or to help our partner troubleshoot their end. I know there are firewalls and switches between me and the remote servers (but I don't know the details except that I should have unrestricted SSH access). I guess I'm thinking there is some network configuration issue between us as the issue is the same for all three servers and it was not like this last week.

2 Answers 2

2

This might be MTU problem. You can quickly verify if it's so with:

ping -M do -s 1472 other.end.address

(1472 = 1500 - 20 (ip header) - 8 (icmp header)).

You can try to narrow down the problem with tracepath. Nowadays, usual things breaking which might cause this type of issues are VPNs/tunnels/etc.

If it's the case, please consider enabling TCP Path MTU Discovery:

sysctl -w net.ipv4.tcp_mtu_probing=1
1
  • Thanks for the suggestion. I've played with this a bit now but I don't think this is it (not too confident though). The ping size that gets through seems to be limited to my source network (1492 at home, 1500 outside). I don't get anything from tracepath that makes me think it is an MTU issue. Anything else we can try?
    – macbutch
    Sep 1, 2014 at 10:58
0

Not very satisfying solution but it has turned out that our partner had an IPS which had been automatically updated with new rules/signatures over the weekend which was causing this behaviour. They have been able to resolve this for us now.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .