On some VMs, it looks like every ssh session ends up as an orphaned process upon exit. I can reproduce it by just login through ssh onto the machine, then doing exit or ^D, then with ps -elf | grep defunct I have one more ssh process. Our monitoring is using ssh a lot, so that's hundreds and hundreds of orphaned zombies by morning, and that on about 20 VMs ..

Here is an exemple of the ps output :

5 Z user  3197     1  0  80   0 -     0 exit   10:00 ?        00:00:00 [sshd] <defunct>

I tried doing a strace of the parent (sshd) to see, here is the output when I exit the ssh session :

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24025, si_status=255, si_utime=0, si_stime=2} ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], WNOHANG, NULL) = 24025
wait4(-1, 0x7ffc0d57901c, WNOHANG, NULL) = 0
rt_sigaction(SIGCHLD, NULL, {0x7f164fee7d70, [], SA_RESTORER, 0x7f164db34d40}, 8) = 0
rt_sigreturn() = -1 EINTR (Interrupted system call)
select(7, [3 4], NULL, NULL, NULL

Not that I really know what I'm looking for in there, but I don't see what could be the problem. Any ideas ? I also see quite a lot of nrpe defunct processes, but restarting sshd cleans up both the ssh and the nrpe zombies, for some reason.

Don't know if that can be relevant, but there is very very slow I/O's on those machines, a simple dd of a few hundreds megabytes takes hundreds of seconds to complete sometimes.

EDIT : As asked, it's ubuntu trusty with openssh 6.6p1-2ubuntu2.7

  • what is "some VMs". OS name, version? OpenSSH version? – Jakuje May 26 '16 at 11:42
  • ubuntu trusty with 6.6p1-2ubuntu2.7 – Ulrar May 26 '16 at 12:20
  • What configuration do you have in /etc/ssh/sshd_config? I am interested in particular in UsePrivilegeSeparation. Sounds like a bug. – Jakuje May 26 '16 at 12:31
  • Seems enabled : UsePrivilegeSeparation yes – Ulrar May 26 '16 at 12:40
  • So It looks like when I log in, two processes are created : a [priv] and a @pts/XX, but only the [priv] gets a wait4, leaving the other one defunct. – Ulrar May 27 '16 at 7:41

So disabling UsePrivilegeSeparation in sshd_config seems to do the trick. Not really a huge fan of doing that, but it works ..

  • Some more comment on this, seems like the root problem wasn't SSH of course, it was upstart. I think it more or less crashed, and wasn't taking care of orphaned processes (not only ssh). Reboot didn't work correctly since the init was kinda crashed, so I had to hard reboot the VM and so far so good, no more problem. Yay for upstart, I guess .. – Ulrar Jun 6 '16 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.