15

After flying from one country to another, I now can't ssh to several of my Digital Ocean Ubuntu servers. However, I can still log in via console and ssh from one box to another (they are all in the same physical data center).

When running ssh with -vvvv and running the time command with it, the last debug message is:

debug2: channel 0: open confirm rwindow 0 rmax 32768
Write failed: Broken pipe

It times out after 1 minute 37 seconds.

Here is the debug log from the point at which ssh key authentication succeeds:

debug1: Authentication succeeded (publickey).
Authenticated to 128.199.170.168 ([128.199.170.168]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env TMPDIR
debug3: Ignored env Apple_PubSub_Socket_Render
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env USER
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env PATH
debug3: Ignored env MARKPATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env XPC_FLAGS
debug3: Ignored env PS1
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env GREP_OPTIONS
debug3: Ignored env LOGNAME
debug3: Ignored env SCALA_HOME
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
Write failed: Broken pipe

The connection isn't particularly slow, my shell is bash (and I can still login via console and other network ssh). Nothing seems to be blocking the ssh connection since I see public key auth taking place.

I don't know which pipe is being written to that is broken. FWIW I'm connecting from OSX but I had no problems until flying to the US.

Here is what auth.log shows when trying to login:

May 17 12:28:01 db1 CRON[24931]: pam_unix(cron:session): session opened for user root by (uid=0)
May 17 12:28:01 db1 CRON[24931]: pam_unix(cron:session): session closed for user root
May 17 12:28:02 db1 sshd[24955]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
May 17 12:28:04 db1 sshd[24955]: Accepted publickey for tomo from 24.210.28.151 port 63202 ssh2: DSA 3a:[redacted]
May 17 12:28:04 db1 sshd[24955]: pam_unix(sshd:session): session opened for user tomo by (uid=0)

Tcpdump capture of port 22 traffic during a connection attempt:

    $ sudo tcpdump -i en0 port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
19:00:40.917870 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [S], seq 3430788632, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 1286503697 ecr 0,sackOK,eol], length 0
19:00:41.211348 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [S.], seq 4135716624, ack 3430788633, win 28960, options [mss 1460,sackOK,TS val 898678531 ecr 1286503697,nop,wscale 8], length 0
19:00:41.211415 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 1, win 4117, options [nop,nop,TS val 1286503989 ecr 898678531], length 0
19:00:41.215051 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 1:22, ack 1, win 4117, options [nop,nop,TS val 1286503992 ecr 898678531], length 21
19:00:41.484824 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [.], ack 22, win 114, options [nop,nop,TS val 898678606 ecr 1286503992], length 0
19:00:41.488532 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 1:42, ack 22, win 114, options [nop,nop,TS val 898678609 ecr 1286503992], length 41
19:00:41.488616 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 42, win 4116, options [nop,nop,TS val 1286504260 ecr 898678609], length 0
19:00:41.490182 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], seq 22:1470, ack 42, win 4116, options [nop,nop,TS val 1286504261 ecr 898678609], length 1448
19:00:41.490183 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 1470:1614, ack 42, win 4116, options [nop,nop,TS val 1286504261 ecr 898678609], length 144
19:00:41.491254 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [.], seq 42:1490, ack 22, win 114, options [nop,nop,TS val 898678609 ecr 1286503992], length 1448
19:00:41.592287 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 1490, win 4096, options [nop,nop,TS val 1286504362 ecr 898678609], length 0
19:00:41.760341 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 1490:1674, ack 22, win 114, options [nop,nop,TS val 898678676 ecr 1286504260], length 184
19:00:41.760401 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 1674, win 4090, options [nop,nop,TS val 1286504527 ecr 898678676], length 0
19:00:41.762375 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [.], ack 1614, win 136, options [nop,nop,TS val 898678676 ecr 1286504261], length 0
19:00:41.762409 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 1614:1638, ack 1674, win 4096, options [nop,nop,TS val 1286504529 ecr 898678676], length 24
19:00:42.027042 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 1674:1826, ack 1638, win 136, options [nop,nop,TS val 898678743 ecr 1286504529], length 152
19:00:42.027103 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 1826, win 4091, options [nop,nop,TS val 1286504789 ecr 898678743], length 0
19:00:42.028104 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 1638:1782, ack 1826, win 4096, options [nop,nop,TS val 1286504790 ecr 898678743], length 144
19:00:42.300304 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 1826:2546, ack 1782, win 148, options [nop,nop,TS val 898678812 ecr 1286504790], length 720
19:00:42.300357 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 2546, win 4073, options [nop,nop,TS val 1286505053 ecr 898678812], length 0
19:00:42.302441 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 1782:1798, ack 2546, win 4096, options [nop,nop,TS val 1286505055 ecr 898678812], length 16
19:00:42.600776 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [.], ack 1798, win 148, options [nop,nop,TS val 898678888 ecr 1286505055], length 0
19:00:42.600843 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 1798:1850, ack 2546, win 4096, options [nop,nop,TS val 1286505349 ecr 898678888], length 52
19:00:42.857852 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [.], ack 1850, win 148, options [nop,nop,TS val 898678952 ecr 1286505349], length 0
19:00:42.858552 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 2546:2598, ack 1850, win 148, options [nop,nop,TS val 898678952 ecr 1286505349], length 52
19:00:42.858584 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 2598, win 4094, options [nop,nop,TS val 1286505604 ecr 898678952], length 0
19:00:42.859131 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 1850:1918, ack 2598, win 4096, options [nop,nop,TS val 1286505605 ecr 898678952], length 68
19:00:43.124310 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 2598:2650, ack 1918, win 148, options [nop,nop,TS val 898679019 ecr 1286505605], length 52
19:00:43.124374 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 2650, win 4094, options [nop,nop,TS val 1286505867 ecr 898679019], length 0
19:00:43.124473 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 1918:2434, ack 2650, win 4096, options [nop,nop,TS val 1286505867 ecr 898679019], length 516
19:00:43.394690 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 2650:2702, ack 2434, win 159, options [nop,nop,TS val 898679086 ecr 1286505867], length 52
19:00:43.394774 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 2702, win 4094, options [nop,nop,TS val 1286506134 ecr 898679086], length 0
19:01:04.685580 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2434:2582, ack 2702, win 4096, options [nop,nop,TS val 1286527239 ecr 898679086], length 148
19:01:04.966270 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 2702:2738, ack 2582, win 170, options [nop,nop,TS val 898684479 ecr 1286527239], length 36
19:01:04.966378 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 2738, win 4094, options [nop,nop,TS val 1286527514 ecr 898684479], length 0
19:01:04.967018 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2582:2702, ack 2738, win 4096, options [nop,nop,TS val 1286527514 ecr 898684479], length 120
19:01:05.269214 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [.], ack 2702, win 170, options [nop,nop,TS val 898684555 ecr 1286527514], length 0
19:01:06.027067 IP [redacted_ip].ssh > 192.168.1.2.50409: Flags [P.], seq 2738:2790, ack 2702, win 170, options [nop,nop,TS val 898684744 ecr 1286527514], length 52
19:01:06.027144 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [.], ack 2790, win 4094, options [nop,nop,TS val 1286528563 ecr 898684744], length 0
19:01:06.027497 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286528563 ecr 898684744], length 460
19:01:06.603432 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286529135 ecr 898684744], length 460
19:01:07.552730 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286530077 ecr 898684744], length 460
19:01:09.250116 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286531762 ecr 898684744], length 460
19:01:12.442790 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286534930 ecr 898684744], length 460
19:01:18.634929 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286541067 ecr 898684744], length 460
19:01:24.068621 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286546451 ecr 898684744], length 460
19:01:34.714519 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286557019 ecr 898684744], length 460
19:01:45.384050 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286567587 ecr 898684744], length 460
19:01:56.051835 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286578155 ecr 898684744], length 460
19:02:06.715163 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286588723 ecr 898684744], length 460
19:02:17.355823 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286599291 ecr 898684744], length 460
19:02:28.042962 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [P.], seq 2702:3162, ack 2790, win 4096, options [nop,nop,TS val 1286609859 ecr 898684744], length 460
19:02:38.690971 IP 192.168.1.2.50409 > [redacted_ip].ssh: Flags [R.], seq 3162, ack 2790, win 4096, length 0

Some other things I've tried:

  • reducing mtu on the server, could be failure in pmtu: sudo ip link set mtu 1280 dev eth0
  • reducing mtu to 1280 in OS X for my wifi interface
  • reducing ServerAliveInterval even lower, to 30, where the connection still times out but not with a broken pipe
  • running ssh with "cat" instead of "bash" or bash but with no profile/rc loaded
  • setting OS X wifi interface IP address manually instead of dhcp
Improve this question
8
  • Obvious explanations include broken NAT, broken firewall, and broken PMTU discovery. Which one of them it is would likely be revealed by a packet trace.
    – kasperd
    May 14, 2015 at 13:18
  • 1
    An MTU of 1480 is not low enough to rule out PMTU discovery problems. Also if you can change the advertised MSS in both directions, then reducing the MSS is a better approach than changing the MTU. An MSS of 1220 will almost always work. When the connection hangs verify that the OS X machine still has the IP address which was used to open the connection. I have seen OS X on some networks first refresh the IP address it had been using previously, and then two minutes later ask the DHCP server for a different IP address.
    – kasperd
    May 15, 2015 at 5:11
  • It would still be a good idea to take a look at a packet capture.
    – kasperd
    May 15, 2015 at 5:43
  • I tried reducing MTU to 1280 on both sides without any results. I appended auth.log output which shows that authentication is succeeding.
    – Tomo Huynh
    May 17, 2015 at 16:33
  • I am still looking for a packet capture.
    – kasperd
    May 17, 2015 at 16:38

5 Answers 5

Reset to default
16

In the packet trace we see that maximum sized packets are exchanged in both directions early on during the flow. That did not cause any problems, so there is nothing suggesting a MTU issue.

Later on during the connection we see that a packet from the client to the server with relative sequence numbers 2702:3162 never receive an ACK from the server.

My immediate thought is that this packet loss is caused by a faulty middlebox (i.e. NAT, firewall, or similar).

I have heard talk about NAT boxes which cannot handle change of TOS during a TCP connection. The problem in your case does happen after the client indicates that TOS has been changed. However since tcpdump doesn't show the TOS, I can't say for sure if that is the exact point where the problem happens.

For a test you can try using -o ProxyCommand='nc %h %p' such that the ssh client doesn't directly control the TCP connection. You can also try the IPQoS option. If the change of TOS is the issue, then specifying -o IPQoS=cs0 or -o IPQoS=0 should work but any other setting would fail. This is because ssh is using 0 as the QoS during authentication and then switch to the chosen QoS after authentication. By choosing QoS to be 0 there won't be any change of QoS value to confuse middleboxes.

Improve this answer
3
  • Thank you, @kasperd. Connecting with "-o ProxyCommand='nc %h %p'" works (as a workaround).
    – Tomo Huynh
    May 19, 2015 at 0:46
  • 1
    @TomoHuynh Nice that ProxyCommand is a functional workaround. I looked a bit more into the IPQoS option and updated my answer with a possible setting for the IPQoS option, which I think will work for you. If changing IPQoS also works for you, then that is a better solution for this scenario than ProxyCommand.
    – kasperd
    May 19, 2015 at 7:03
  • Running with -o IPQoS=0, as suggested by @kasperd, worked for me as well.
    – Marc Tamsky
    Apr 22 at 23:47
9

In case anybody else comes across this, I had a similar issue with an TP-Link Archer VR2600 router/modem (with firmware 1.4.0 0.8.0 v0050.0 Build 160518 Rel.50944n).

Running with -o IPQoS=0, as suggested by @kasperd, worked suggesting some sort of QoS issue with my router. I enabled the closest thing I could find in the router settings (Advanced → Bandwidth Control setting max bandwidth just below what's available on my line) under the assumption that the router might start paying attention to the relevant flags in this case.

This seemed to work and my connections are now getting through. Toggling this option reliably controls whether I can get through.

Improve this answer
1
  • hey @sam-mason, did you ever find a more permanent solution? I have the same exact modem. For now I'm using the "ProxyCommand" workaround. This is frustrating as this isn't really a cheap router.
    – leonardoborges
    Mar 5, 2017 at 0:25
0

Do you have a user ssh config (~/.ssh/config)?

If not create one and try adding the following lines to it:

ServerAliveInterval 120 #ping the server every 120s
TCPKeepAlive no #do not set SO_KEEPALIVE on socket
Improve this answer
3
  • I do have a ssh config, and it already had the ServerAliveInterval setting. I added the TCPKeepAlive setting (and found out the ssh client doesn't accept comments on the same line so removed them) to no avail. I want to make clear that it's not a problem of connecting normally then timing out after inactivity. The problem is that the connection is almost made but at the end doesn't actually launch me into a shell.
    – Tomo Huynh
    May 14, 2015 at 22:53
  • Do you have access to the auth log or sshd logging on the remote system? Also as it was mentioned above 1480 is not low enough to rule out an MTU problem.
    – sclarson
    May 15, 2015 at 13:44
  • Yes, auth.log shows me logging in. I tried lowering MTU down to 1280 too.
    – Tomo Huynh
    May 17, 2015 at 16:35
0

I have also a tp-link router (C9). Direct connection from my laptop to my (internet) server are sometimes not working or hanging, but when i connect to another server in the same network my laptop is in i have no problems. Laptop is always connected via wlan, my internal server directly to the router.

unfortunately -o IPQoS=0 and -o ProxyCommand='nc %h %p' was not helping.

So i looked at my router settings and disabled "NAT boost" and viola it was working perfectly with direct connections. To Verify it was really that setting i enabled it again and: still working :-(

I suspect that the reboot of the router "fixed" the problem for me. So far it is running for 3 days and didn't had a problem connecting or hanging connections as before.

I now have a script running testing the connection all the time and will update my findings when i have the problems again. You might want to test if a simple reboot of the router helps you too.

Improve this answer
0

Sadly I don't have enough reputation here to vote up or comment on Sam Mason's response above, but I would just like to publicly +1 what he said. I also have a VR2600, and I also had the same experience:

  1. Connection (ssh, sftp, etc) gets made but then just seems to hang
  2. tshark shows TCP Spurious Retransmits
  3. setting -o IPQoS=0 (on its own) from client side didn't do anything
  4. enabling the Advanced->Bandwidth Control setting on the router (previously disabled) , with the highest limits possible (= effectively unlimited), seems to fix the router such that it pays attention to the IPQoS flag
  5. tshark no longer shows TCP Spurious Retransmits and connection no longer hangs (ssh,sftp,etc clients now function with a server behind the VR2600 router).

It seems to suggest a significant bug in the VR2600 router. Unfortunately (as of writing), I am on the latest firmware (1.4.0 0.8.0 v0050.0 Build 160518 Rel.50944n , same as Sam) , and this router does not seem to be compatible with / tested with DD-WRT .

HOWEVER, to add to what was discussed above, I will also say:

  1. Having performed steps 1 thru 5, I can now successfully connect even without specifying "-o IPQoS=0"

In other words:

Merely turning on the Advanced->Bandwidth Control option in the router options (even with a maximally high upper limit) seems to be sufficient to get this router NAT to behave as expected. If bandwidth control is disabled, the problem as described in the OP (and in good detail by @malasa) occurs.

Unclear if the remedy is to simply enable this option, or if you need to have connected at least once with the -o option too. In any case, I can confirm that, after having enabled this option, if I then disable that Advanced->Bandwidth Control option, then my ssh/sftp/etc are broken as before. If I enable that Advanced->Bandwidth Control option, everything appears to work as expected again. And (having enabled this option) everything appears to work fine across router reboots also.

So it's a pretty good workaround/fix that requires no client-side changes or maintenance, from my point of view (to answer @leonardoborges concern)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.