Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a number of CentOS 5.x and RHEL 5.x systems whose SSH daemons become unresponsive. This prevents remote logins.

The typical error from the connecting side is:

$ ssh db1
db1 :  ssh_exchange_identification: Connection closed by remote host

Examining /var/log/messages after a forced reboot shows the following leading up to the restart:

Dec 10 10:45:51 db1 sshd[14593]: fatal: Privilege separation user sshd does not exist
Dec 10 10:46:02 db1 sshd[14595]: fatal: Privilege separation user sshd does not exist
Dec 10 10:46:54 db1 sshd[14711]: fatal: Privilege separation user sshd does not exist
Dec 10 10:47:38 db1 sshd[14730]: fatal: Privilege separation user sshd does not exist

These systems use LDAP authentication and the nsswitch.conf file is configured to look at local "files" first.

[root@db1 ~]# cat /etc/nsswitch.conf
#
# /etc/nsswitch.conf
#

passwd:     files ldap
shadow:     files ldap
group:      files ldap

hosts:      files dns

The Privilege-separated SSH user exists in the local password file.

[root@db1 ~]# grep ssh /etc/passwd
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin

Any ideas on what the root cause is? I did not see any Red Hat errata that covers this.

share|improve this question
    
Does it also exist in /etc/shadow? –  Michael Hampton Dec 10 '12 at 17:55
    
@MichaelHampton Yes, "sshd" is in /etc/shadow. –  ewwhite Dec 10 '12 at 18:00
    
Is there either a sshd user in LDAP with a conflicting UID, and/or another user in LDAP with UID 74? Try both getent passwd sshd and getent passwd 74 when the SSHD is in a broken state. –  David Mackintosh Dec 10 '12 at 20:11
    
@DavidMackintosh I can't get into the systems during this period to check. –  ewwhite Dec 10 '12 at 20:17
    
Hmmm... well maybe set a cron job to run them regularly (every 15 minutes maybe) and then pipe them into the syslog? eg: (getent passwd sshd ; getent passwd 74) | logger -p warn -t getent -- and then check to make sure it is getting logged in /var/messages. –  David Mackintosh Dec 10 '12 at 22:13

2 Answers 2

up vote 1 down vote accepted

Debian bug #552431 sounds very similar.

Do the affected systems do more LDAP queries than non-affected systems? E.g., mail servers, LDAP-authenticated DB servers?

nss-ldap in EL5 is not well designed, it was replaced by nss-pam-ldapd in EL6. Do you have any EL6 machines with or without this problem?

If the problem is reproducible and you have the ability to experiment, I suggest trying sssd to replace nss_ldap and nscd. SSSD is in the RHEL/CentOS repos, available with yum. Note: sssd does not cache hosts like nscd does, if you need to cache hostnames when using sssd you should use a caching DNS server (dnsmasq is super easy for that) or use nscd to only cache hosts. SSSD does cache user/passwd/group info.

share|improve this answer
    
The affected systems are Oracle database servers. I'm in the process of getting my EL5 systems in this environment to use SSSD, as well as converting all of the EL6 systems to SSSD. The latter have terrible memory leak issues. –  ewwhite Dec 11 '12 at 21:22

SUGGESTION 1: Does /var/empty/sshd exist? It should be a root owned 755 permission directory.

SUGGESTION 2: Any error in the /etc/passwd file? (i.e. have you run pwck?)

SUGGESTION 3: Try terminating the nscd daemon process. nscd has some bugs and sometimes the caching with passwd, group, or host information does not properly work.

SUGGESTION 4: Does /sbin/nologin exist?

share|improve this answer
    
I see: drwx--x--x 3 root root 4096 Jan 4 2012 /var/empty/sshd/. –  ewwhite Dec 10 '12 at 17:50
    
pwck output is clean. nscd is only running on some of the affected systems. –  ewwhite Dec 10 '12 at 18:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.