Best Practices for Linux Daemons

Question

I've been looking up information on running background processes on Linux. The basic requirements are:

• Start the process and keep it running no matter what happens
• If the process quits for whatever reason, immediately relaunch it
• Be able to launch the process as a specific user

Additionally, some processes need to be run not continuously, but periodically. This is of course a job for cron, and more than likely it's what I'll use for that. But for continuous processes, it's a different story.

On Mac OS X, I use launchd for this purpose. Launchd is the canonical method for launching processes on Mac OS X. The operating system uses it for ALL system services, so I have no qualms about relying on it. So far I haven't found a similar system for Linux. Various distros all seem to have their own methods (Ubuntu's upstart, etc.) and I still can't find a straight answer on how /etc/rc.local, /etc/init.d/rc.local, /etc/initd/rc.local, or /etc/init all relate and what the difference is between them. They're different on every system. Add to that the fact that "service start/stop x" doesn't work on all systems (and Google hasn't helped here) and you have what I'd call a mess.

What I want is ONE place I can save my scripts where I can reboot the system and have them launched as daemons at boot (before any user logs in), then have them automatically restarted if they ever terminate. And I want it to be rock solid reliable and fully compatible with multiple distros, not some afterthought like upstart.

Is this even possible?

Answer 1

Sorry, the Linux distributions (in their infinite quest to differentiate themselves and make life difficult for system administrators) have each standardized on doing the startup thing differently.

The closest thing to a "cross-distribution standard" for Linux startup scripts would be the System V style runlevels. These are deprecated on some systems (for example Ubuntu's official position is you should use Upstart), but they still work and probably will for a good long while.

SysV init scripts do not handle "automatically restarting" if your daemon crashes - they launch a daemon which is expected to take care of itself.
If you want supervision you would need to write/include your own supervisor framework.
(Alternatively simply write a decent piece of software that doesn't crash, which is the traditional Unix way of dealing with this. If your daemon is so unstable that it needs extra supervision to keep restarting it that usually means you need to do more work on making it production-quality.)

Answer 2

You were close with upstart - what you need is systemd. It is not fully supported yet, but handles old-style SysV init scripts (though without supervision) and can be easily adopted to run on many distros. There is no better solution available for now out of the box, but if you prefer not to change entire init you might be interested in monit daemon, which can be easily instructed to supervise almost any service.

Answer 3

What I want is ONE place I can save my scripts where I can reboot the system and have them launched as daemons at boot (before any user logs in), then have them automatically restarted if they ever terminate. And I want it to be rock solid reliable and fully compatible with multiple distros, not some afterthought like upstart.

This doesn't exist. Even if you limit yourself to the enterprise linux distributions, you'll find a variety of init systems, e.g.

• SysVInit: Red Hat Enterprise Linux 5, SUSE Linux Enterprise 11
• Upstart: Red Hat Enterprise Linux 6
• Systemd: Red Hat Enterprise Linux 7, SUSE Linux Enterprise 12

The most "compatible" way is to write sysvinit scripts, since both upstart and systemd support them. However, it doesn't support automatically restarting services if they die. Both upstart and systemd do support that.