0

I have done a lot of searching on this issue, and can not find anything particularly relevant.

I have many instances spread across zones and grouped into various load balancers. One of the groups is an "API" utilized by various different applications for common and secure tasks.

In the API group we have one instance that uses an Elastic IP so that we can reliably use crons for tasks such as rsync etc. Someone before my time also decided it would be a good idea to hard code this IP into various applications in typical spaghetti code if else forest fashion... long story short, very important instance.

Two days ago I was suddenly unable to SSH to this guy (It was created over a year ago). The pem key works across ALL other instances, which have been around for relatively the same amount of time.

This is output from failed connection on home machine (yesterday) attempts using ssh -v -i path/file.pem user@ip

ssh -v -i <path>/<file>.pem <user>@<ip>                                                                                             
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011                                                        
debug1: Reading configuration data /etc/ssh/ssh_config                                                      
debug1: Applying options for *                                                                                      
debug1: Connecting to <ip> [<ip>] port 22.                                              
debug1: Connection established.                                                                                     
debug1: identity file /home/<user>/.ssh/id_rsa type -1                                                          
debug1: identity file /home/<user>/.ssh/id_rsa-cert type -1                                                 
debug1: identity file /home/<user>/.ssh/id_dsa type -1                                                          
debug1: identity file /home/<user>/.ssh/id_dsa-cert type -1
debug1: identity file /home/<user>/.ssh/id_ecdsa type -1
debug1: identity file /home/<user>/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 75:43:6d:03:43:f5:89:fa:8d:fe:64:e1:39:9a:73:26
debug1: Host '<ip>' is known and matches the ECDSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:78
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: .ssh/<file>.pem
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/<user>/.ssh/id_rsa
debug1: Trying private key: /home/<user>/.ssh/id_dsa
debug1: Trying private key: /home/<user>/.ssh/id_ecdsa
debug1: No more authentication methods to try.
Permission denied (publickey).

Finally after searching for hours on what to do about this I ran ssh -Tvvv -i path/file.pem user@ip and magically I had a debug connection to the instance.

This was done from a machine at home. I killed the connection, and connected normally. No problems. Connected from a server at the office with no problem.

Today I can not SSH into the instance from my machine at the office, and all converted keys do not work on this instance either (i.e. ppk keys for filezilla etc.)

Here is -v debug output from a working machine (home machine)

ssh -v -i <path>/<file>.pem <user>@<ip>
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to <ip> [<ip>] port 22.
debug1: Connection established.
debug1: identity file /home/<user>/.ssh/<file>.pem type -1
debug1: identity file /home/<user>/.ssh/<file>.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 75:43:6d:03:43:f5:89:fa:8d:fe:64:e1:39:9a:73:26
debug1: Host '<ip>' is known and matches the ECDSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:78
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: .ssh/<file>.pem
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/<user>/.ssh/<file>.pem
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to <ip> ([<ip>]:22).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: Remote: Ignored authorized keys: bad ownership or modes for directory /home/<user>
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Welcome to blah...

All permissions on keys are identical across instances and machines connecting to instances.

Does anyone have any idea what can be done to correct this?

Thanks, -David

--EDIT--

This is the -v debug output of work machine that still will not establish a SSH connection.

I also recently cleared the known hosts on this machine.

ssh -v -i <path>/<file>.pem <user>@<ip>
OpenSSH_6.0p1 Debian-3ubuntu1, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to <ip> [<ip>] port 22.
debug1: Connection established.
debug1: identity file /home/<user>/.ssh/<file>.pem type -1
debug1: identity file /home/<user>/.ssh/<file>.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-3ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 75:43:6d:03:43:f5:89:fa:8d:fe:64:e1:39:9a:73:26
debug1: Host '<ip>' is known and matches the ECDSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/<user>/.ssh/<file>.pem
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

I would like to reiterate that all converted keys are not working for sftp despite whether an ssh connection can be established. Specifically .ppk key converted to be used in filezilla. The same follows that the .ppk works on all other instances except the problem child... and also worked previously.

--EDIT 2--

The machine I was able to connect with was restarted, and I can no longer connect with it.

3
  • Are those two ssh logs from the same local machine? There are differences in the log that I can't see being from the remote. Jan 22, 2013 at 21:30
  • Yes, they are actually. I cat the debug output yesterday before the -Tvvv magic fix and had not deleted it yet. The 'good' connection was run while I was typing up this question. Both from home machine.
    – David
    Jan 22, 2013 at 21:35
  • I edited the question to specify this, and added output from my machine that is still not able to establish a connection.
    – David
    Jan 22, 2013 at 22:00

2 Answers 2

3

Here is your problem:

debug1: Remote: Ignored authorized keys: bad ownership or modes for directory /home/<user>

Your user's home directory may not be group or world writable, and the .ssh directory and .ssh/authorized_keys may be readable only by the user. Once you fix the permissions, you should find your logins working again.

1
  • Did not occur to me to check the /home directory. Thank you.
    – David
    Feb 5, 2013 at 6:14
0

In my experience, I learned to kill misbehaving ec2 instances with extreme prejudice. As your key works on your other instances and not on this particular instance, I would chalk it up as a messed up instance. If you really need the data on that server you could always shut it down and mount the ebs volume onto another instance and from there perform recovery.

4
  • I wish that was an option. We can not shutdown this instance for that long. Thank you for your input though.
    – David
    Jan 22, 2013 at 22:13
  • At some point, if it's a bad instance, it's going to shut down for good, whether you can handle the downtime or not.
    – ceejayoz
    Feb 2, 2013 at 18:26
  • 2
    @David EC2 instances are intended to be used as disposable building blocks. You need to be using identically configured instances in an autoscaling group for anything important.
    – jamieb
    Feb 2, 2013 at 18:30
  • I am well aware of this. I am not responsible for the initial architecture of this mess.
    – David
    Feb 5, 2013 at 6:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .