2

I have a functioning EC2 instance with several users, some of whom are chrooted to their home directories, some of whom are ftp-only and have no shell access, etc... ec2-user is the main admin account, though others also have root access and full ssh logins. Everything works great on the running instance.

I can take a snapshot image of the instance and launch new instances from the snapshot. No matter what I select in terms of the key pair associated with the new instance (use the original keypair for ec2-user, create a new keypair, or use no keypair), once the new instance is launched and running, I am unable to ssh into the server using ec2-user or any other ssh-enabled user. ftp works fine, however.

Security groups are not an issue, as far as I can tell, the incoming traffic is allowed (and it's the same security group as the original instance, anyways).

The /var/log/secure of the login attempts gives me:

sshd[1739]: debug1: userauth-request for user ec2-user service ssh-connection method none
sshd[1739]: debug1: attempt 0 failures 0
sshd[1738]: debug1: user ec2-user does not match group list sftponly at line 142
sshd[1738]: debug1: PAM: initializing for "ec2-user"
sshd[1738]: debug1: PAM: setting PAM_RHOST to "..."
sshd[1738]: debug1: PAM: setting PAM_TTY to "ssh"
sshd[1739]: debug1: userauth-request for user ec2-user service ssh-connection method publickey
sshd[1739]: debug1: attempt 1 failures 0
sshd[1738]: debug1: temporarily_use_uid: 500/500 (e=0/0)
sshd[1738]: debug1: trying public key file /etc/ssh/keys/ec2-user
sshd[1738]: debug1: restore_uid: 0/0
sshd[1738]: debug1: temporarily_use_uid: 500/500 (e=0/0)
sshd[1738]: debug1: trying public key file /etc/ssh/keys/ec2-user
sshd[1738]: debug1: restore_uid: 0/0
sshd[1738]: Failed publickey for ec2-user from xx.xx.xx.xx port 60597 ssh2
sshd[1739]: Connection closed by xx.xx.xx.xx 
sshd[1739]: debug1: do_cleanup

It's the same error for all ssh-enabled users. As you can see from the log, I had changed my sshd_config on the original instance so that it looks for the public keys in the /etc/ssh/keys folder.

I have mounted the failed instances as volumes on the working instance. The keys are in the folder, with the same permissions, and the same key values, all as expected. Here is ls -al of the keys folder (.) and the ec2-user file. 

drwxr-xr-x. 2 root     root     4096 Dec  1 16:59 .
-rw-------. 1 ec2-user ec2-user  388 Jul 25 13:27 ec2-user

What could be causing this problem? I would like to solve the problem at the point of saving and launching a snapshot or in setting up the original instance, but not mounting the problematic instance and making manual changes so that it is functional but doesn't fix the larger problem.

UPDATE: Here are the active settings in the sshd_config file:

#...
Protocol 2
#...
SyslogFacility AUTHPRIV
LogLevel DEBUG
#...
AuthorizedKeysFile /etc/ssh/keys/%u
#...
PasswordAuthentication no
#...
ChallengeResponseAuthentication no
#...
GSSAPIAuthentication yes
#...
GSSAPICleanupCredentials yes
#...
UsePAM yes
#...
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
#...
X11Forwarding yes
#...
Subsystem       sftp    internal-sftp -f AUTH -l VERBOSE
#...
Match group sftponly
ChrootDirectory /home/%u
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp -l VERBOSE -f AUTH
#...
Improve this question
5
  • What Flavour of Linux is this?
    – GeoSword
    Dec 12, 2013 at 10:22
  • Red Hat Enterprise Linux Server release 6.5 (Santiago)
    – cheepychappy
    Dec 12, 2013 at 17:03
  • I Suspect this is an SELinux problem. Put your SELinux into permissive mode and see if the problem goes away, if it does, youll need to reset the security context. Ill dig out the documentation I have on that
    – GeoSword
    Dec 12, 2013 at 17:07
  • If SELinux is the problem, are you saying to put the original machine into permissive mode before taking the snapshot?
    – cheepychappy
    Dec 12, 2013 at 17:13
  • I think you'll have to, because I guess you have no means of connecting in its current state.
    – GeoSword
    Dec 12, 2013 at 17:40

2 Answers 2

Reset to default
1

I suspect this is an SELinux problem. Check the context of the folder you are using, I expect it won't be correct for holding keys.

Im afraid I no longer have access to a Redhat box to establish exactly what the context should be. That said, try this:

ls -lZ /root/.ssh

This will yield the selinux context that your new folder needs to be. If I remember correctly it should be something like system_u:system_r:sshd_t

Then we need to apply that security context to your new authorized keys location:

semanage fcontext -a -t system_u:system_r:sshd_t “/etc/ssh/keys(/.*)?”

Which associated the correct context with the new keys location. Finally, we can apply the context to the new location

restorecon -Rv /etc/ssh/keys
Improve this answer
9
  • Thanks GeoSword, this looks promising. Here's what your first command showed: ls -lZ /root/.ssh -rw-------. root root system_u:object_r:ssh_home_t:s0 authorized_keys -rw-r--r--. root root unconfined_u:object_r:ssh_home_t:s0 known_hosts But your fcontext command wouldn't accept the : syntax as given, so I googled it and ended up just running semanage fcontext -a -t system_u:object_r:ssh_home_t:s0 “/etc/ssh/keys(/.*)?” I tested to make sure I could still log in to the original system and it checked out. Sadly, when I went to create the image/reboot, I got an unrelated kernel panic!
    – cheepychappy
    Dec 12, 2013 at 23:12
  • I'll let you know if I have success after dealing with the kernel issue (lesson learned: don't run yum update on EC2). But I have the feeling you are on the right track... I had some SELinux issues when I originally set up the accounts on the machine.
    – cheepychappy
    Dec 12, 2013 at 23:18
  • Sorry, first comment was copied from the wrong terminal window. Command I ended up running was: semanage fcontext -a -t ssh_home_t "/etc/ssh/keys(/.*)?"
    – cheepychappy
    Dec 12, 2013 at 23:19
  • Update: Fixed the kernel issue, tried to boot a new instance from the fixed image, which already had the fcontext and restorecon changes in place, was not able to get in. Changed SELinux to permissive, saved a new image, launched an instance. Still not able to get in. Looking at the /var/log/secure files for each of those instances, they both show a "permission denied" error: Could not open authorized keys '/etc/ssh/keys/ec2-user': Permission denied Which for whatever reason is different from the original error but still confusing as key ownership/permissions are identical to original.
    – cheepychappy
    Dec 13, 2013 at 7:45
  • One somewhat-interesting piece of information. AWS is adding a new root file to the /etc/ssh/keys folder of the new instances. Context begins system_u rather than unconfined_u, owned by root:root and contains the same private key as ec2-user. I figure this is created when I select the pre-existing key pair when launching the instance. So I guess AWS has some awareness of my sshd settings and puts the key in the right spot? That is surprising to me (forgive my lack of expertise). Sadly, AWS forbids ssh-ing as root, so can't test to see if this key works.
    – cheepychappy
    Dec 13, 2013 at 7:55
0

Is the 'AllowUsers' directive set in sshd_config ? Maybe its set to a specific user on the original server and ssh has not been restarted yet, so it still accepts all users?

Oh I just saw this in the debug: "user ec2-user does not match group list sftponly at line 142" do check sshd_config you might have disallowed a group, or allowed only sftp?

Improve this answer
1
  • The sftponly group is for users that should NOT have shell access (only sftp), so it is a good thing that the ssh user is not matching that group. I'll add the sshd_config settings to the original post if you want to see for yourself.
    – cheepychappy
    Dec 12, 2013 at 17:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .