1

I have a number of CentOS 5.x and RHEL 5.x systems whose SSH daemons become unresponsive. This prevents remote logins.

The typical error from the connecting side is:

$ ssh db1
db1 :  ssh_exchange_identification: Connection closed by remote host

Examining /var/log/messages after a forced reboot shows the following leading up to the restart:

Dec 10 10:45:51 db1 sshd[14593]: fatal: Privilege separation user sshd does not exist
Dec 10 10:46:02 db1 sshd[14595]: fatal: Privilege separation user sshd does not exist
Dec 10 10:46:54 db1 sshd[14711]: fatal: Privilege separation user sshd does not exist
Dec 10 10:47:38 db1 sshd[14730]: fatal: Privilege separation user sshd does not exist

These systems use LDAP authentication and the nsswitch.conf file is configured to look at local "files" first.

[root@db1 ~]# cat /etc/nsswitch.conf
#
# /etc/nsswitch.conf
#

passwd:     files ldap
shadow:     files ldap
group:      files ldap

hosts:      files dns

The Privilege-separated SSH user exists in the local password file.

[root@db1 ~]# grep ssh /etc/passwd
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin

Any ideas on what the root cause is? I did not see any Red Hat errata that covers this.

6
  • Does it also exist in /etc/shadow? Dec 10, 2012 at 17:55
  • @MichaelHampton Yes, "sshd" is in /etc/shadow.
    – ewwhite
    Dec 10, 2012 at 18:00
  • Is there either a sshd user in LDAP with a conflicting UID, and/or another user in LDAP with UID 74? Try both getent passwd sshd and getent passwd 74 when the SSHD is in a broken state. Dec 10, 2012 at 20:11
  • @DavidMackintosh I can't get into the systems during this period to check.
    – ewwhite
    Dec 10, 2012 at 20:17
  • Hmmm... well maybe set a cron job to run them regularly (every 15 minutes maybe) and then pipe them into the syslog? eg: (getent passwd sshd ; getent passwd 74) | logger -p warn -t getent -- and then check to make sure it is getting logged in /var/messages. Dec 10, 2012 at 22:13

3 Answers 3

1

Debian bug #552431 sounds very similar.

Do the affected systems do more LDAP queries than non-affected systems? E.g., mail servers, LDAP-authenticated DB servers?

nss-ldap in EL5 is not well designed, it was replaced by nss-pam-ldapd in EL6. Do you have any EL6 machines with or without this problem?

If the problem is reproducible and you have the ability to experiment, I suggest trying sssd to replace nss_ldap and nscd. SSSD is in the RHEL/CentOS repos, available with yum. Note: sssd does not cache hosts like nscd does, if you need to cache hostnames when using sssd you should use a caching DNS server (dnsmasq is super easy for that) or use nscd to only cache hosts. SSSD does cache user/passwd/group info.

1
  • The affected systems are Oracle database servers. I'm in the process of getting my EL5 systems in this environment to use SSSD, as well as converting all of the EL6 systems to SSSD. The latter have terrible memory leak issues.
    – ewwhite
    Dec 11, 2012 at 21:22
1

Just as reminder. It happens to me too:

sshd[*]: fatal: Privilege separation user sshd does not exist

I made verry basic mistake during SAMBA setup:

passwd:     winbind
group:      winbind
shadow:     files

Should be at least:

passwd:     files winbind
group:      files winbind
shadow:     files

smbcontrol all reload-config

Missing information about files cause not taking information from local files (as far as I understand). I think that this wrong setup affect user nobody and thus causing problems.

PS: Feel free to correct me :)

0

SUGGESTION 1: Does /var/empty/sshd exist? It should be a root owned 755 permission directory.

SUGGESTION 2: Any error in the /etc/passwd file? (i.e. have you run pwck?)

SUGGESTION 3: Try terminating the nscd daemon process. nscd has some bugs and sometimes the caching with passwd, group, or host information does not properly work.

SUGGESTION 4: Does /sbin/nologin exist?

2
  • I see: drwx--x--x 3 root root 4096 Jan 4 2012 /var/empty/sshd/.
    – ewwhite
    Dec 10, 2012 at 17:50
  • pwck output is clean. nscd is only running on some of the affected systems.
    – ewwhite
    Dec 10, 2012 at 18:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .