I have a functioning EC2 instance with several users, some of whom are chrooted to their home directories, some of whom are ftp-only and have no shell access, etc... ec2-user is the main admin account, though others also have root access and full ssh logins. Everything works great on the running instance.

I can take a snapshot image of the instance and launch new instances from the snapshot. No matter what I select in terms of the key pair associated with the new instance (use the original keypair for ec2-user, create a new keypair, or use no keypair), once the new instance is launched and running, I am unable to ssh into the server using ec2-user or any other ssh-enabled user. ftp works fine, however.

Security groups are not an issue, as far as I can tell, the incoming traffic is allowed (and it's the same security group as the original instance, anyways).

The /var/log/secure of the login attempts gives me:

sshd[1739]: debug1: userauth-request for user ec2-user service ssh-connection method none
sshd[1739]: debug1: attempt 0 failures 0
sshd[1738]: debug1: user ec2-user does not match group list sftponly at line 142
sshd[1738]: debug1: PAM: initializing for "ec2-user"
sshd[1738]: debug1: PAM: setting PAM_RHOST to "..."
sshd[1738]: debug1: PAM: setting PAM_TTY to "ssh"
sshd[1739]: debug1: userauth-request for user ec2-user service ssh-connection method publickey
sshd[1739]: debug1: attempt 1 failures 0
sshd[1738]: debug1: temporarily_use_uid: 500/500 (e=0/0)
sshd[1738]: debug1: trying public key file /etc/ssh/keys/ec2-user
sshd[1738]: debug1: restore_uid: 0/0
sshd[1738]: debug1: temporarily_use_uid: 500/500 (e=0/0)
sshd[1738]: debug1: trying public key file /etc/ssh/keys/ec2-user
sshd[1738]: debug1: restore_uid: 0/0
sshd[1738]: Failed publickey for ec2-user from xx.xx.xx.xx port 60597 ssh2
sshd[1739]: Connection closed by xx.xx.xx.xx 
sshd[1739]: debug1: do_cleanup

It's the same error for all ssh-enabled users. As you can see from the log, I had changed my sshd_config on the original instance so that it looks for the public keys in the /etc/ssh/keys folder.

I have mounted the failed instances as volumes on the working instance. The keys are in the folder, with the same permissions, and the same key values, all as expected. Here is ls -al of the keys folder (.) and the ec2-user file.

drwxr-xr-x. 2 root     root     4096 Dec  1 16:59 .
-rw-------. 1 ec2-user ec2-user  388 Jul 25 13:27 ec2-user

What could be causing this problem? I would like to solve the problem at the point of saving and launching a snapshot or in setting up the original instance, but not mounting the problematic instance and making manual changes so that it is functional but doesn't fix the larger problem.

UPDATE: Here are the active settings in the sshd_config file:

Protocol 2
SyslogFacility AUTHPRIV
LogLevel DEBUG
AuthorizedKeysFile /etc/ssh/keys/%u
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding yes
Subsystem       sftp    internal-sftp -f AUTH -l VERBOSE
Match group sftponly
ChrootDirectory /home/%u
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp -l VERBOSE -f AUTH
  • What Flavour of Linux is this? – GeoSword Dec 12 '13 at 10:22
  • Red Hat Enterprise Linux Server release 6.5 (Santiago) – cheepychappy Dec 12 '13 at 17:03
  • I Suspect this is an SELinux problem. Put your SELinux into permissive mode and see if the problem goes away, if it does, youll need to reset the security context. Ill dig out the documentation I have on that – GeoSword Dec 12 '13 at 17:07
  • If SELinux is the problem, are you saying to put the original machine into permissive mode before taking the snapshot? – cheepychappy Dec 12 '13 at 17:13
  • I think you'll have to, because I guess you have no means of connecting in its current state. – GeoSword Dec 12 '13 at 17:40

I suspect this is an SELinux problem. Check the context of the folder you are using, I expect it won't be correct for holding keys.

Im afraid I no longer have access to a Redhat box to establish exactly what the context should be. That said, try this:

ls -lZ /root/.ssh

This will yield the selinux context that your new folder needs to be. If I remember correctly it should be something like system_u:system_r:sshd_t

Then we need to apply that security context to your new authorized keys location:

semanage fcontext -a -t system_u:system_r:sshd_t “/etc/ssh/keys(/.*)?”

Which associated the correct context with the new keys location. Finally, we can apply the context to the new location

restorecon -Rv /etc/ssh/keys
  • Thanks GeoSword, this looks promising. Here's what your first command showed: ls -lZ /root/.ssh -rw-------. root root system_u:object_r:ssh_home_t:s0 authorized_keys -rw-r--r--. root root unconfined_u:object_r:ssh_home_t:s0 known_hosts But your fcontext command wouldn't accept the : syntax as given, so I googled it and ended up just running semanage fcontext -a -t system_u:object_r:ssh_home_t:s0 “/etc/ssh/keys(/.*)?” I tested to make sure I could still log in to the original system and it checked out. Sadly, when I went to create the image/reboot, I got an unrelated kernel panic! – cheepychappy Dec 12 '13 at 23:12
  • I'll let you know if I have success after dealing with the kernel issue (lesson learned: don't run yum update on EC2). But I have the feeling you are on the right track... I had some SELinux issues when I originally set up the accounts on the machine. – cheepychappy Dec 12 '13 at 23:18
  • Sorry, first comment was copied from the wrong terminal window. Command I ended up running was: semanage fcontext -a -t ssh_home_t "/etc/ssh/keys(/.*)?" – cheepychappy Dec 12 '13 at 23:19
  • Update: Fixed the kernel issue, tried to boot a new instance from the fixed image, which already had the fcontext and restorecon changes in place, was not able to get in. Changed SELinux to permissive, saved a new image, launched an instance. Still not able to get in. Looking at the /var/log/secure files for each of those instances, they both show a "permission denied" error: Could not open authorized keys '/etc/ssh/keys/ec2-user': Permission denied Which for whatever reason is different from the original error but still confusing as key ownership/permissions are identical to original. – cheepychappy Dec 13 '13 at 7:45
  • One somewhat-interesting piece of information. AWS is adding a new root file to the /etc/ssh/keys folder of the new instances. Context begins system_u rather than unconfined_u, owned by root:root and contains the same private key as ec2-user. I figure this is created when I select the pre-existing key pair when launching the instance. So I guess AWS has some awareness of my sshd settings and puts the key in the right spot? That is surprising to me (forgive my lack of expertise). Sadly, AWS forbids ssh-ing as root, so can't test to see if this key works. – cheepychappy Dec 13 '13 at 7:55

Is the 'AllowUsers' directive set in sshd_config ? Maybe its set to a specific user on the original server and ssh has not been restarted yet, so it still accepts all users?

Oh I just saw this in the debug: "user ec2-user does not match group list sftponly at line 142" do check sshd_config you might have disallowed a group, or allowed only sftp?

  • The sftponly group is for users that should NOT have shell access (only sftp), so it is a good thing that the ssh user is not matching that group. I'll add the sshd_config settings to the original post if you want to see for yourself. – cheepychappy Dec 12 '13 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.