2

My company is moving to a new office. I regularly access AWS EC2 instances via SSH/SFTP from the office. I am unable to ssh to an EC2 instance from the new office from a Linux machine. I CAN ssh to the SAME instance from a windows machine (via PuTTy) in the old office (using the same RSA key/username).

I suspect that the problem lies in a misconfiguration of our outbound network in the new office. That would be my IT guy's problem. I'm trying to understand, however, if it is really MY problem!

The old office used Century Link. The new office uses Comcast.

Here are the gory details from 'ssh -vvv' from the new office to the EC2 instance (I am getting the 'Permission denied (publickey,gssapi-keyex,gssapi-with-mic)' error without any detail):

ssh -vvv -o StrictHostKeyChecking=no -i /home/mbrown/.ssh/2019_11_13_AWS_Oregon.pem  [email protected]
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /home/mbrown/.ssh/config
debug1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 ec2-18-236-161-133.us-west-2.compute.amazonaws.com
debug3: Not a RSA1 key file /home/mbrown/.ssh/2019_11_13_AWS_Oregon.pem.
debug1: permanently_drop_suid: 501
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/mbrown/.ssh/2019_11_13_AWS_Oregon.pem type -1
debug1: identity file /home/mbrown/.ssh/2019_11_13_AWS_Oregon.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version PaloAltoNetworks_0.2
debug1: no match: PaloAltoNetworks_0.2
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 864 bytes for a total of 885
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,[email protected]
debug2: kex_parse_kexinit: hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96
debug2: kex_parse_kexinit: hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: found hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 909
debug2: dh_gen_key: priv key bits set: 171/320
debug2: bits set: 1024/2048
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 272 bytes for a total of 1181
debug3: check_host_in_hostfile: host ec2-18-236-161-133.us-west-2.compute.amazonaws.com filename /home/mbrown/.ssh/known_hosts
debug3: check_host_in_hostfile: host ec2-18-236-161-133.us-west-2.compute.amazonaws.com filename /home/mbrown/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 2
debug1: Host 'ec2-18-236-161-133.us-west-2.compute.amazonaws.com' is known and matches the RSA host key.
debug1: Found key in /home/mbrown/.ssh/known_hosts:2
debug2: bits set: 1006/2048
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1197
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 52 bytes for a total of 1249
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/mbrown/.ssh/2019_11_13_AWS_Oregon.pem ((nil))
debug3: Wrote 68 bytes for a total of 1317
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_501' not found

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_501' not found

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/mbrown/.ssh/2019_11_13_AWS_Oregon.pem
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA f5:a4:12:94:bf:d1:4d:6c:94:4d:ed:a0:38:85:6c:1d:6c:7b:08:b6
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 644 bytes for a total of 1961
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I dug into the sshd server logs on the EC2 instance. Here is what happens when I attempt to ssh from the new office:

Nov 14 17:31:06 ip-172-31-19-149 sshd[3350]: error: key_verify: incorrect signature
Nov 14 17:31:06 ip-172-31-19-149 sshd[3350]: Connection closed by 50.220.29.106 port 62192 [preauth]

Here is the output of a successful ssh from the windows machine in the old office:

Nov 14 17:33:25 ip-172-31-19-149 sshd[3407]: reverse mapping checking getaddrinfo for 50-224-166-10-static.hfc.comcastbusiness.net [50.224.166.10] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 14 17:33:26 ip-172-31-19-149 sshd[3407]: Accepted publickey for ec2-user from 50.224.166.10 port 60874 ssh2: RSA SHA256:lmkjrzq7H2aA/VBcE2CKsG8V7zPqeMUCgrKEpZ7Ndxc
Nov 14 17:33:26 ip-172-31-19-149 sshd[3407]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)

From the EC2 instance, I cannot ping the external IP of the new office (50.220.29.106), but I CAN ping the external IP of the old office (50.224.166.10)

As for some of the obvious (e.g. from EC2 troubleshooting) questions...

1) Permissions on my local key file are OK

-r--------. 1 mbrown mbrown 1670 Nov 14 11:05 2019_11_13_AWS_Oregon.pem

2) I am using the correct ssh user (ec2-user)

3) Yes, the 2019_11_13_AWS_Oregon keyname is associated with the EC2 instance

Improve this question
11
  • 1
    This sounds like a Network ACL or Security Group problem on the AWS side. Have you checked and verified that they're correctly configured since the move to the new office?
    – joeqwerty
    Nov 15, 2019 at 19:31
  • My security group is wide open (shame on me!?). Yesterday we had active network connections at BOTH the old and new offices. So I was able to successfully ssh to the AWS instance via a windows machine. At the same time, I could not successfully ssh to the AWS instance via a Linux server.
    – Morgan Brown
    Nov 16, 2019 at 0:07
  • you got a response from ec2 machine so I don't think there is any network issue in your office's outbound or at aws side, its something related to the ssh key, what you can try is that create another ssh key-pair and add its public key into ec2 user's authorized_keys and then try ssh with the new key, see if that helps
    – Deepak Deore
    Nov 16, 2019 at 3:08
  • Are you using the same machine in both offices? It sounds like maybe your SSH key might not be setup correctly in the new office.
    – Adam Luchjenbroers
    Nov 16, 2019 at 3:28
  • Thanks. I tried what you said and unfortunately get the same error. FYI the "authorized_keys" file on the AWS instance DOES have an entry from my most recent unsuccessful attempt.
    – Morgan Brown
    Nov 16, 2019 at 3:28

3 Answers 3

Reset to default
1

This seems to be a "smart firewall issue". The SSH traffic seems to be intercepted, unpacked, and terminated by a "smart firewall" rule.

Nowadays, this is a common policy preventing potential malicious (malware) from using SSH.

So ask your network administrator for details.

Improve this answer
0

Looks indeed as this was an internal problem. We have a third party handling configuration of our firewall. My IT thought that our firewall was "wide open", but apparently the firewall was actually set to restrict inbound traffic from Amazon. The network tech added "exceptions" for my Linux servers.

Improve this answer
0

Ensure that your new office IP Address is allowed in security group to SSH.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .