I have an issue which I am struggling to understand. I have a server which is hosted by a business parter on their network but my team is responsible for installing and configuring software to this server (actually there are 3 servers all exhibiting the same behaviour). As part of the configuration we are trying to copy files via rsync (over SSH) but we are running into issues which we and out partner do not understand.
Essentially it seems we are able to rsync files that are less than 32768 bytes but after that the connection stalls. We are doing this over SSH and I can get a responsive shell on the server. I am connecting across the internet but I have tried from two locations with the same results. Here is an example of what I see:
rsync -aP ~/Downloads/file.zip servername:~ -vvv
opening connection using ssh servername rsync --server -vvvlogDtpr --partial . "~"
building file list ...
[sender] make_file(file.zip,*,2)
1 file to consider
server_recv(2) starting pid=2610
send_file_list done
send_files starting
received 1 names
recv_file_list done
get_local_name count=1 /home/me
generator starting pid=2610
delta-transmission enabled
recv_files(1) starting
recv_generator(file.zip,0)
send_files(0, /Users/me/Downloads/file.zip)
send_files mapped /Users/me/Downloads/file.zip of size 54965002
calling match_sums /Users/me/Downloads/file.zip
file.zip
32768 0% 0.00kB/s 0:00:00
This will stall for some minutes and eventually timeout. I have captured the packets from my side while I'm not particularly versed in reading packet captures it seems that the server side stops responding for a few minutes and eventually resets the connection. I found a tshark snippet on another question which I tweaked slightly to get this:
tshark -r ~/rsync-timeout.pcap -q -z io,stat,20,"COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission","COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack","COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment","COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission"
===================================================================================
| IO Statistics |
| |
| Duration: 395.924510 secs |
| Interval: 20 secs |
| |
| Col 1: COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission |
| 2: COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack |
| 3: COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment |
| 4: COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission |
|---------------------------------------------------------------------------------|
| |1 |2 |3 |4 | |
| Interval | COUNT | COUNT | COUNT | COUNT | |
|--------------------------------------------| |
| 0 <> 20 | 0 | 0 | 0 | 0 | |
| 20 <> 40 | 2 | 1 | 0 | 0 | |
| 40 <> 60 | 13 | 0 | 0 | 0 | |
| 60 <> 80 | 4 | 0 | 0 | 0 | |
| 80 <> 100 | 4 | 0 | 0 | 0 | |
| 100 <> 120 | 0 | 0 | 0 | 0 | |
| 120 <> 140 | 4 | 0 | 0 | 0 | |
| 140 <> 160 | 0 | 0 | 0 | 0 | |
| 160 <> 180 | 0 | 0 | 0 | 0 | |
| 180 <> 200 | 4 | 0 | 0 | 0 | |
| 200 <> 220 | 0 | 0 | 0 | 0 | |
| 220 <> 240 | 4 | 0 | 0 | 0 | |
| 240 <> 260 | 0 | 0 | 0 | 0 | |
| 260 <> 280 | 4 | 0 | 0 | 0 | |
| 280 <> 300 | 0 | 0 | 0 | 0 | |
| 300 <> 320 | 4 | 0 | 0 | 0 | |
| 320 <> 340 | 0 | 0 | 0 | 0 | |
| 340 <> 360 | 4 | 0 | 0 | 0 | |
| 360 <> 380 | 0 | 0 | 0 | 0 | |
| 380 <> Dur | 0 | 0 | 0 | 0 | |
===================================================================================
I can see that that isn't great but I'm not really sure what it tells me. I can see in the packet trace that there is no response from the server (or none getting to me) for a few minutes and then finally a RST is set and both sides hang up.
The end of my packet trace as viewed with tshark looks like this:
... everything seems ok before this point
429 45.647732 1.2.3.4 -> 192.168.1.18 TCP 66 22→53839 [ACK] Seq=2438 Ack=15846 Win=60288 Len=0 TSval=8701862 TSecr=552453169
430 45.714775 1.2.3.4 -> 192.168.1.18 TCP 66 22→53839 [ACK] Seq=2438 Ack=17254 Win=63232 Len=0 TSval=8701928 TSecr=552453237
431 45.748600 1.2.3.4 -> 192.168.1.18 TCP 66 22→53839 [ACK] Seq=2438 Ack=18662 Win=64128 Len=0 TSval=8701963 TSecr=552453237
432 45.821013 1.2.3.4 -> 192.168.1.18 TCP 66 22→53839 [ACK] Seq=2438 Ack=21478 Win=64128 Len=0 TSval=8702035 TSecr=552453237
433 47.331298 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
434 49.243984 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
435 49.243989 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
436 49.244199 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
437 49.244203 192.168.1.18 -> 1.2.3.4 SSHv2 882 Client: [TCP Retransmission] , Encrypted packet (len=816)
438 52.678673 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
439 52.678677 192.168.1.18 -> 1.2.3.4 SSHv2 1254 Client: [TCP Retransmission] , Encrypted packet (len=1188)
... more of the same ...
472 309.358046 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
473 309.358166 192.168.1.18 -> 1.2.3.4 SSHv2 222 Client: [TCP Retransmission] , Encrypted packet (len=156)
474 335.714554 1.2.3.4 -> 192.168.1.18 TCP 60 22→53837 [RST, ACK] Seq=4050 Ack=5018 Win=0 Len=0
475 352.579591 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
476 352.579592 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
477 352.579595 192.168.1.18 -> 1.2.3.4 SSHv2 1474 Client: [TCP Retransmission] , Encrypted packet (len=1408)
478 352.579596 192.168.1.18 -> 1.2.3.4 SSHv2 222 Client: [TCP Retransmission] , Encrypted packet (len=156)
479 395.924510 192.168.1.18 -> 1.2.3.4 TCP 54 53839→22 [RST, ACK] Seq=29014 Ack=2438 Win=131072 Len=0
I would love to have some ideas on what we can do to troubleshoot this or to help our partner troubleshoot their end. I know there are firewalls and switches between me and the remote servers (but I don't know the details except that I should have unrestricted SSH access). I guess I'm thinking there is some network configuration issue between us as the issue is the same for all three servers and it was not like this last week.