I have jenkins and gitosis running on the same [Ubuntu 10.04] server. Jenkins fetches git commits from a repo served by gitosis (also running on the same server). Many repos, actually, since there are many jobs in Jenkins. Every now and then I end up with a zombie
who process, where the root cause appears to be some Ubuntu-specific scripts to update /etc/motd. Here's what (part of) the process table looked like earlier today:
jenkins 30042 1 0 Mar19 ? 00:00:00 git fetch -t email@example.com:testRepo.git +refs/heads/*:refs/remotes/origin/* jenkins 30060 30042 0 Mar19 ? 00:00:00 ssh firstname.lastname@example.org git-upload-pack 'testRepo.git' root 30066 703 0 Mar19 ? 00:00:00 sshd: gitosis [priv] root 30096 30066 0 Mar19 ? 00:00:00 sh -c /usr/bin/env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /bin/run-parts --lsbsysinit /etc/update-motd.d > /var/run/motd.new root 30097 30096 0 Mar19 ? 00:00:00 /bin/run-parts --lsbsysinit /etc/update-motd.d root 30129 30097 0 Mar19 ? 00:00:00 /bin/sh /etc/update-motd.d/50-landscape-sysinfo root 30148 30129 0 Mar19 ? 00:00:00 /usr/bin/python /usr/bin/landscape-sysinfo root 30451 30148 0 Mar19 ? 00:00:00 [who] <defunct>
To clear it up I basically just killed every parent of the zombie
who, then the zombie disappeared.
Any ideas on how to prevent this from happening or gracefully deal with it when it does?
Is there some way to disable these motd scripts sometimes? Like, only when Jenkins is doing
git fetch? Actually, what would be perfect is if I could somehow disable the motd scripts whenever the gitosis user tries to SSH in (e.g. whenever any git client/server interaction takes place).
Maybe there's a straightforward way with Bash to kill any git process left running for over an hour? Ugh.
I generally like the handy motd provided by these scripts so I don't want to completely disable them. Sure would be nice if they'd timeout on their own. Why the heck is the motd updated on every login? Once a day seems like plenty.
I don't really want to switch to gitolite or a different git server, gitosis otherwise works fine. I don't want to switch to another protocol besides SSH; gitosis relies on SSH. Since gitosis is running on the same server, I should be able to refer to the git repositories using
file:/// syntax. I'll give that a shot now. Update... nope, that won't work since that bypasses gitolite and won't allow jenkins to push tags back upstream. I could give jenkins more ACLs, but I'd rather stick with gitolite.
Side note: this won't help since it appears the problem occurs after the SSH connection is established.