"Sometimes" I get "Permission denied (publickey)"

Question

Context: ssh:ing to my ssh server from somewhere with key (i.e. not passwd), and sometimes it doesn't work.

Server: Ubuntu 16.04 LTS, fully patched. OpenSSH server. Passwords and root login disabled.

Clients: Tried with OSX OpenSSH client and Ubuntu 17.10 OpenSSH client.

Trailing output of ssh -vvv "server" when failing (from OSX):

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xxxx/E /Users/xxx/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51 <-- HERE, diff from success
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
xxx@yyy.zzz: Permission denied (publickey).

Trailing output of ssh -vvv "server" when succeeding (from OSX, this one a couple of min later than the failing one above):

debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:xxx/E /Users/xxx/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60 <-- HERE, success, diff from fail
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input_userauth_pk_ok: fp SHA256:xxx/E
debug3: sign_and_send_pubkey: RSA SHA256:xxx/E
Enter passphrase for key '/Users/xxx/.ssh/id_rsa': 
debug1: identity added to agent: /Users/xxx/.ssh/id_rsa
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to yyy.zzz ([123.456.789.012]:22).

And they are quite similar, the difference is at this line:

debug3: receive packet: type 51 (when failing)
debug3: receive packet: type 60 (when succeeding)

This leads me to believe it is a server side issue. In /var/log/auth.log I find these entries:

Feb  7 09:30:59 server-name sshd[48527]: Connection closed by (client public IP) port 64050 [preauth] (the only mention of this connection attempt)
Feb  7 09:34:17 server-name sshd[48725]: Accepted publickey for yyy from (client public IP) port 64134 ssh2\: RSA SHA256:xxx/E (the succeeding attempt)

So there is something happening, but now I'm stumped? Any ideas on how to solve this?

Could be relevant info: The ssh server has a public IP, and there are roughly ten (10) bad ssh connection attempts a minute (only port 22 is open). It seems that for a couple of minutes after I've logged onto the server locally it is always possible to log on remote via ssh. The server sits behind a physical firewall, with port 22 forwarded, and the behaviour is the same from my local subnet.

Answer 1

After messing around a bit I've found the issue to be connected to an encrypted home-directory (which I completely missed during setup since I was setting up 10+ VMs through scripts).

Still confusing that the server did not log that it was not able to access /home/userdir/.ssh/authorized_keys, and only showed:

Feb  7 09:30:59 server-name sshd[48527]: Connection closed by (client public IP) port 64050 [preauth] (the only mention of this connection attempt)

In general there are two solutions:

1. Decrypt the home directory (this is messy, I would not recommend it). Google it for instructions. ecryptfs permanent decrypt gets good results.
2. Move the authorized_keys outside of your encrypted home folder so it is accessible.

Since 1) is messy I would recommend 2).

Moving authorized_keys

I would recommend creating a directory structure under /etc/ssh/ like /etc/ssh/keys/%user/authorized_keys and changing the AuthorizedKeyFile line in /etc/ssh/sshd_config to match. I.e.:

#original (%h expands to /home/userdir, which is encrypted)
AuthorizedKeysFile     %h/.ssh/authorized_keys
#new (%u expands to username)
AuthorizedKeysFile     /etc/ssh/keys/%u/authorized_keys

After logging in now you should be in a minimalistic home-folder with no contents, run ecryptfs-mount-private to decrypt home-folder (you will need to enter passphrase, by default your password). Easiest way to get around this is adding a .profile in your minimalistic home-folder which decrypts and sends you on to your real home-folder.

# place in minimalistic .profile
ecryptfs-mount-private
# if below doesn't work, replace with static cd /home/userdir
cd $HOME