I have a functioning EC2 instance with several users, some of whom are chrooted to their home directories, some of whom are ftp-only and have no shell access, etc... ec2-user is the main admin account, though others also have root access and full ssh logins. Everything works great on the running instance.
I can take a snapshot image of the instance and launch new instances from the snapshot. No matter what I select in terms of the key pair associated with the new instance (use the original keypair for ec2-user, create a new keypair, or use no keypair), once the new instance is launched and running, I am unable to ssh into the server using ec2-user or any other ssh-enabled user. ftp works fine, however.
Security groups are not an issue, as far as I can tell, the incoming traffic is allowed (and it's the same security group as the original instance, anyways).
The /var/log/secure of the login attempts gives me:
sshd: debug1: userauth-request for user ec2-user service ssh-connection method none sshd: debug1: attempt 0 failures 0 sshd: debug1: user ec2-user does not match group list sftponly at line 142 sshd: debug1: PAM: initializing for "ec2-user" sshd: debug1: PAM: setting PAM_RHOST to "..." sshd: debug1: PAM: setting PAM_TTY to "ssh" sshd: debug1: userauth-request for user ec2-user service ssh-connection method publickey sshd: debug1: attempt 1 failures 0 sshd: debug1: temporarily_use_uid: 500/500 (e=0/0) sshd: debug1: trying public key file /etc/ssh/keys/ec2-user sshd: debug1: restore_uid: 0/0 sshd: debug1: temporarily_use_uid: 500/500 (e=0/0) sshd: debug1: trying public key file /etc/ssh/keys/ec2-user sshd: debug1: restore_uid: 0/0 sshd: Failed publickey for ec2-user from xx.xx.xx.xx port 60597 ssh2 sshd: Connection closed by xx.xx.xx.xx sshd: debug1: do_cleanup
It's the same error for all ssh-enabled users. As you can see from the log, I had changed my sshd_config on the original instance so that it looks for the public keys in the /etc/ssh/keys folder.
I have mounted the failed instances as volumes on the working instance. The keys are in the folder, with the same permissions, and the same key values, all as expected. Here is ls -al of the keys folder (.) and the ec2-user file.
drwxr-xr-x. 2 root root 4096 Dec 1 16:59 . -rw-------. 1 ec2-user ec2-user 388 Jul 25 13:27 ec2-user
What could be causing this problem? I would like to solve the problem at the point of saving and launching a snapshot or in setting up the original instance, but not mounting the problematic instance and making manual changes so that it is functional but doesn't fix the larger problem.
UPDATE: Here are the active settings in the sshd_config file:
#... Protocol 2 #... SyslogFacility AUTHPRIV LogLevel DEBUG #... AuthorizedKeysFile /etc/ssh/keys/%u #... PasswordAuthentication no #... ChallengeResponseAuthentication no #... GSSAPIAuthentication yes #... GSSAPICleanupCredentials yes #... UsePAM yes #... AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE AcceptEnv XMODIFIERS #... X11Forwarding yes #... Subsystem sftp internal-sftp -f AUTH -l VERBOSE #... Match group sftponly ChrootDirectory /home/%u X11Forwarding no AllowTcpForwarding no ForceCommand internal-sftp -l VERBOSE -f AUTH #...