6

Having managed to kill sshd on a remote machine (by running a script which used all available memory in the machine, oops...) which I have no access to other than visiting the hosting for[1], I was considering ways to ensure that sshd is always kept running.

Other than a hacky cron job to restart sshd every n minutes or hours, using inittab to get init to keep sshd running seems like a good idea.

Are there any drawbacks from this approach? It would seem like something which it would be sensible for Linux distros to do by default, since sshd is often the only available method of access for a machine..

Additionally, are there any other daemons which I should be using this approach for? Perhaps a monitoring agent such as nrpe for nagios?

[1] Yes, management cards or a network power switch would be a good idea, but they were deemed "unnecessary" at the time...

Improve this question
3
  • Ah, it seems that there is the terrible drawback that it won't work at all, due to sshd backgrounding itself (thanks katriel!). Disabling this is only possible via the -d (debug) option, which sets it to only one connection as well as putting it in the foreground. I'll have to go with an alternative init such as Upstart, runit or daemontools as suggested by a few people. At least until sshd gets an option to run in every way as a daemon but not to set itself into the background...
    – David Gardner
    Jul 24, 2009 at 20:25
  • 1
    Good grief, the -D option (immediately above -d in the man page) does exactly what you want.
    – womble ♦
    Jul 24, 2009 at 23:25
  • Doh! How did I miss that?... I'll certainly try this :)
    – David Gardner
    Jul 25, 2009 at 9:21

8 Answers 8

Reset to default
3

There's a few implementations of this idea. Upstart is used by Ubuntu and can restart services if they die, Solaris 10 has the Service Management Facility, runit is cross-platform and there's daemontools as already mentioned.

Improve this answer
1
  • To be more precise: Fedora uses upstart as well
    – wzzrd
    Jul 24, 2009 at 12:34
2

I can think of no other reason not to do the inittab thing, than that restarting sshd after an upgrade is a little more inconvenient.

Other than that: interesting idea.

Improve this answer
3
  • Must be some reason why it's not done... security issues? user levels? can you run apps as anything other than root?
    – sybreon
    Jul 24, 2009 at 13:19
  • @sybreon, well, SSH does privilege separation, so after login, sshd spawns a process just for you under your account, even though the mother ssh process runs as root. Security shouldn't be an issue any more than it would be for agetty. Not sure what you mean by 'user levels'. And: hi! You here too? :-)
    – wzzrd
    Jul 24, 2009 at 13:39
  • @sybreon, my guess is it is much less convenient to restart sshd in this setup, so after an update, you are stuck with the old version (rpms tend to restart this service on update, debs too probably). Only way to restart it would be to kill it. On a sidenote I have never noticed an sshd crash.
    – wzzrd
    Jul 24, 2009 at 13:42
2

you can tell linux OOM killer not to kill sshd, google for oom_adj for more details, or see i.e. here rhel manual

Improve this answer
1
  • 'RTFM' is considered a bad answer, one of the reasons being that links break over time. Please post an example.
    – aggregate1166877
    Mar 10, 2021 at 3:22
1

There are benefits to having services that need to be reliable under a scheme that will ensure they're always running. I prefer to use daemontools, myself, for the reasons documented here: http://cr.yp.to/daemontools/faq/create.html

I've not run ssh this way, but I would be happy enough doing it if I was in the situation where I thought that my current SSH management wouldn't work. As far as your "running out of memory" problem, you can deprioritise certain processes like sshd so that they don't get killed by the OOM killer in favour of the program actually causing the problem.

Improve this answer
2
  • 1
    Any idea how you de-prioritise processes like sshd?
    – Cian
    Jul 24, 2009 at 12:16
  • 2
    @Cian You alter the value in /proc/$PID/oom_adj, lower score -> decrease likelyhood of it being killed, higher-score -> increase likelyhood of it being killed
    – wzzrd
    Jul 24, 2009 at 12:33
1

An interesting idea.

I haven't tried anything like that but I would check at what time in the boot process the things from inittab are started. If it is too early you may not have the network running.

Improve this answer
1
  • Depends what runlevel it is set to run at, so this should be OK.
    – David Gardner
    Jul 25, 2009 at 9:24
1

Monit is a monitoring daemon that is designed for just what you want to do here.

Improve this answer
0

The only issue that I can foresee is if it were attempt to respawn with a configuration error.

I thought you could rate limit respawning, but I can't seem to find any documentation to support this.

Improve this answer
0

Like other have noted before me, using an existing tool like daemontools or monit will probably be the smartest route. You can't use inittab to spawn sshd wince it forks to the background, and init will try to run several sshd's. You'll most likely get "init: re-spawning too fast" messages.

You might want to write a small monitoring script that will run in a loop and make sure the original sshd (the one that accepts connections and forks to handle the sessions) is still running. One it fails, just use the system's init script to re-run it.

Just a note, if your sshd is killed by the kernel's OOM handler, there'e no guarantee your sshd will survive a restart...

Improve this answer
1
  • You can tell sshd not to background on start, and daemontools relies on that behaviour to work properly.
    – womble ♦
    Jul 24, 2009 at 23:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .