Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My client is Ubuntu, and the server is CentOS. I have shared key authentication set up to login from client to server. When I run the server like:

sudo `which sshd` -d

Everything works as expected; I don't get a user login prompt. I take this to mean that everything is in the correct place, with correct permissions, etc. When I restart the server in "normal" mode:

sudo service sshd restart

My client prompts me for the user password.

Here is verbose output from success (sshd -d):

anregen@ba-vm-atf-01:~/workspace$ ssh -v web@test.lab.network.com
OpenSSH_6.0p1 (CentrifyDC build 5.1.1-814) (CentrifyDC build 5.1.1-814), OpenSSL 0.9.8w 23 Apr 2012
debug1: Reading configuration data /etc/centrifydc/ssh/ssh_config
debug1: /etc/centrifydc/ssh/ssh_config line 52: Applying options for *
debug1: Connecting to test.lab.network.com [172.20.80.45] port 22.
debug1: Connection established.
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_rsa type 1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_rsa-cert type -1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_dsa type -1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_dsa-cert type -1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_ecdsa type -1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0
debug1: Miscellaneous failure
No credentials cache found

debug1: Miscellaneous failure
No credentials cache found

debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 1e:57:9c:69:d2:09:29:0b:5b:4d:e4:f7:98:ee:f5:94
debug1: Host 'test.lab.network.com' is known and matches the RSA host key.
debug1: Found key in /home/MyNetwork.local/anregen/.ssh/known_hosts:7
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Miscellaneous failure
No credentials cache found

debug1: Miscellaneous failure
No credentials cache found

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/MyNetwork.local/anregen/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to test.lab.network.com ([172.20.80.45]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
Last login: Tue Jun 17 16:03:39 2014 from ba-vm-atf-01.lab.network.com
Environment:
  USER=web
  LOGNAME=web
  HOME=/var/www
  PATH=/usr/local/bin:/bin:/usr/bin
  MAIL=/var/mail/web
  SHELL=/bin/bash
  SSH_CLIENT=172.20.80.27 32973 22
  SSH_CONNECTION=172.20.80.27 32973 172.20.80.45 22
  SSH_TTY=/dev/pts/1
  TERM=xterm
  SELINUX_ROLE_REQUESTED=
  SELINUX_LEVEL_REQUESTED=
  SELINUX_USE_CURRENT_RANGE=
-bash-4.1$

And here is the verbose output from a "failure" sudo service sshd restart:

anregen@ba-vm-atf-01:~/workspace$ ssh -v web@test.lab.network.com
OpenSSH_6.0p1 (CentrifyDC build 5.1.1-814) (CentrifyDC build 5.1.1-814), OpenSSL 0.9.8w 23 Apr 2012
debug1: Reading configuration data /etc/centrifydc/ssh/ssh_config
debug1: /etc/centrifydc/ssh/ssh_config line 52: Applying options for *
debug1: Connecting to test.lab.network.com [172.20.80.45] port 22.
debug1: Connection established.
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_rsa type 1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_rsa-cert type -1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_dsa type -1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_dsa-cert type -1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_ecdsa type -1
debug1: identity file /home/MyNetwork.local/anregen/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0
debug1: Miscellaneous failure
No credentials cache found

debug1: Miscellaneous failure
No credentials cache found

debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 1e:57:9c:69:d2:09:29:0b:5b:4d:e4:f7:98:ee:f5:94
debug1: Host 'test.lab.network.com' is known and matches the RSA host key.
debug1: Found key in /home/MyNetwork.local/anregen/.ssh/known_hosts:7
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Miscellaneous failure
No credentials cache found

debug1: Miscellaneous failure
No credentials cache found

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/MyNetwork.local/anregen/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/MyNetwork.local/anregen/.ssh/id_dsa
debug1: Trying private key: /home/MyNetwork.local/anregen/.ssh/id_ecdsa
debug1: Next authentication method: password
web@test.lab.network.com's password:

Any thoughts on what is going on with -d that makes this work? Or is it something with running sshd outside of the service tool?

EDIT: By request, here are server-side logs with -d option. I added #comments to show the remote/client activity.

[anregen@ba-vm-web-01 ~]$ clear
[anregen@ba-vm-web-01 ~]$ sudo service sshd stop
Stopping sshd:                                             [  OK  ]
[anregen@ba-vm-web-01 ~]$ sudo `which sshd` -d
debug1: sshd version OpenSSH_5.3p1
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.

# At this point, I log in from remote as "web"
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 172.20.80.27 port 37828
debug1: Client protocol version 2.0; client software version OpenSSH_6.0
debug1: match: OpenSSH_6.0 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: permanently_set_uid: 74/74
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user web service ssh-connection method none
debug1: attempt 0 failures 0
debug1: PAM: initializing for "web"
debug1: PAM: setting PAM_RHOST to "ba-vm-atf-01.lab.network.com"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user web service ssh-connection method publickey
debug1: attempt 1 failures 0
debug1: test whether pkalg/pkblob are acceptable
debug1: temporarily_use_uid: 500/500 (e=0/0)
debug1: trying public key file /var/www/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /var/www/.ssh/authorized_keys, line 1
Found matching RSA key: 6b:b4:a1:41:e2:7c:0a:c6:2b:6d:27:87:f4:3c:6e:0c
debug1: restore_uid: 0/0
Postponed publickey for web from 172.20.80.27 port 37828 ssh2
debug1: userauth-request for user web service ssh-connection method publickey
debug1: attempt 2 failures 0
debug1: temporarily_use_uid: 500/500 (e=0/0)
debug1: trying public key file /var/www/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /var/www/.ssh/authorized_keys, line 1
Found matching RSA key: 6b:b4:a1:41:e2:7c:0a:c6:2b:6d:27:87:f4:3c:6e:0c
debug1: restore_uid: 0/0
debug1: ssh_rsa_verify: signature correct
debug1: do_pam_account: called
Accepted publickey for web from 172.20.80.27 port 37828 ssh2
debug1: monitor_child_preauth: web has been authenticated by privileged process
debug1: temporarily_use_uid: 500/500 (e=0/0)
debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
debug1: restore_uid: 0/0
debug1: SELinux support enabled
debug1: PAM: establishing credentials
debug1: temporarily_use_uid: 500/500 (e=0/0)
debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
debug1: restore_uid: 0/0
User child is on pid 19221
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 500/500
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_new: session 0
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
debug1: Setting controlling tty using TIOCSCTTY.

# at this point I send "exit" from remote to logout
debug1: Received SIGCHLD.
debug1: session_by_pid: pid 19222
debug1: session_exit_message: session 0 channel 0 pid 19222
debug1: session_exit_message: release channel 0
debug1: session_by_tty: session 0 tty /dev/pts/1
debug1: session_pty_cleanup: session 0 release /dev/pts/1
Received disconnect from 172.20.80.27: 11: disconnected by user
debug1: do_cleanup
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials
share|improve this question
2  
What was logged on the server? –  Michael Hampton Jun 17 at 16:30
2  
Do you have encrypted home directories? –  MadHatter Jun 17 at 16:37
    
Please excuse my ignorance here: I do not have any sort of .encryptfs files or similar in my home directories. Is that conclusive that the dirs are not encrypted? –  anregen Jun 17 at 19:03
    
I have no idea, I'm afraid I'm not ubuntu expert. But encrypted home directories would produce exactly the result you're seeing, assuming that the sudo sshd -d was started by the same user that you were trying to ssh in as. –  MadHatter Jun 17 at 19:40
    
it isn't the same user. On the client machine: "anregen", on the server machine: "web". The user that sudo on the server is also anregen. I have not set up any shared key association between the two anregen accounts though. –  anregen Jun 17 at 20:09

1 Answer 1

up vote 2 down vote accepted

TLDR; SELinux

I went back to using the same user account on both machines. More importantly, this means I am using home directories that SELinux is on-board with. You can see in the logs that the "web" user has non-standard home directory /var/www/ (it is intended to be the web content mgmt account)

I then removed the ~/.ssh/ directory on the server machine, the went back over to the client and used ssh-copy-id. That apparently recreated the ~/.ssh/ directory and auth files in a SELinux-compatible way. Note that simply removing and recreating the .ssh directory for the "web" user was not sufficient. It seems that when I changed the "web" user home directory, SELinux was not properly involved...

I don't have an answer as to why -d changes this behavior on the server-side, but if anyone else bumps into this, try removing the ~.ssh/ directory, and letting the ssh-copy-id script do the work for you.

update Apparently, because running it as sudo changes the security permissions for the process. So, even though a ps looks the same, I found that ps -Z (which is the SELinux-context-aware argument) does not.

The only thing left at this point is for me to research how to make a user with home directory /var/www in the SELinux-approved way so that it respects that accounts authorized_keys file

update I used chcon -R --reference=/home/anregen/.ssh /var/www/.ssh. That was the trick that got the correct security context into the relocated "home" directory for user: web. Keep in mind this is sort of temporary, meaning you should not just use chcon, but rather make a new policy.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.