0

I have a group of Ubuntu 14.04 workstations that are all clones deployed via image. After imaging, they are assigned a static IP and they also delete/regenerate their host keys. Each of them on their own seems fine, they can connect to the file server, computing cluster, etc. just fine, but they can't connect to each other via SSH, and just hang endlessly (until they time out after ~10 minutes) on debug1: SSH2_MSG_KEXINIT sent.

I can connect to them all just fine via a Windows laptop, or an older 12.04 workstation, or a 12.04 server, or from one problem workstation to another by hopping through anything else, but not like-to-like.

  • maybe cloned machines having same MAC address or something? Is it just SSH that is failing between the cloned machines or does other networking fail also. Can they ping each other? If you run netcat to listen on a port like nc -l 3333 and then try to telnet to it from another cloned server telnet server.ip 3333 does that work ok ? – Flo Woo Aug 11 '14 at 5:49
  • Netcat works, HTTP works, ifconfig reports different MAC addresses, could they be inadvertently spoofed somehow (but would that have the SSH symptom)? From what I can tell with SSH, they do successfully contact one another, it's just at that particular step (host key exchange?) something goes awry. – Nick T Aug 11 '14 at 5:57
  • 1
    fire up tcpdump/wireshark and perform a capture. – Zoredache Aug 11 '14 at 6:38
2

I was able to reproduce the symptom by introducing a PMTU black hole in my network connectivity.

On Ubuntu 12.04 the key exchange init message is about 1KB in size, which is well within the typical network MTU. On Ubuntu 14.04 the key exchange init message has grown to almost 2KB in size making it the first message during the connection to exceed a typical network MTU.

This means the symptom of an MTU issue will have changed. Earlier you may have been able to connect, but transferring a large file over the connection or running a command which produce lots of output quickly, could stall the connection. But now an MTU issue cause the connection to stall before authentication.

Reducing the MTU on the network interface is a usable workaround until you identify the root cause. It can also be used to confirm that the problem is indeed MTU related. If the network interface is called eth0, you could try this command:

ifconfig eth0 mtu 1280
  • Lowering the MTU does cure the problem, but does that suggest this is a switch or other infrastructure problem or something local on the machines? – Nick T Aug 11 '14 at 15:19
  • @NickT In order to figure out where the MTU problem is located, you will have to start looking at packet captures like Zoredache suggests. The only time I have seen an MTU problem on a switch, was when I had configured a network to send tagged frames over an unmanaged switch. Turns out it could forward tagged frames just fine, but only as long as the packet including tag was at most 1500 bytes. It is more likely the problem is on a router or locally on a machine. – kasperd Aug 11 '14 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.