How to automatically restart a service on failure in Linux

Question

On Windows, you can set what should happen if/when a service fails. Is there a standard way of achieving the same thing on Linux (CentOS in particular)?

A bigger part of my question is: how do you handle sockets that have been left open - for example in TIME_WAIT, FIN_WAIT1, etc states.

At the moment if the service I am developing crashes, I have to wait for the sockets to clear or change the listen port before I can then manually restart it.

Thanks for your help.

Answer 1

monit is a great way to monitor and restart services when they fail--and you'll probably end up using this for other essential services (such as Apache). There's a nice article on nixCraft detailing how to use this for services specifically, although monit itself has many more functions beyond this.

As for the socket aspect, @galraen answered this spot on.

Answer 2

Only answering the service restart part. I came across Monit as well, but on CentOS 7 systemd takes care of all that for you. You just need to add these two lines to the .service file (if they're not there already):

Restart=always
RestartSec=3

Run man systemd.service for reference.

If you want to create a custom systemd service, it's pretty straightforward to write your own service file. See the example below, for a custom http server.

Start the editor with a new service file:

vim /etc/systemd/system/httpd.service

And add the following content, which you can edit as required:

[Unit]
Description=My httpd Service
After=network.target

[Service]
Type=simple
User=root
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PERLLIB=/perl
ExecStart=/bin/httpd /etc/httpd.conf
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

I want it to start automatically on boot:

systemctl enable httpd

Tell systemd about the changes and start the service:

systemctl daemon-reload
systemctl start httpd

And now you can see the status:

systemctl status httpd

There are many options for Restart. The following is an excerpt from the man page:

Restart=

Configures whether the service shall be restarted when the service process exits, is killed, or a timeout is reached...

Takes one of no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always. If set to no (the default), the service will not be restarted. If set to on-success, it will be restarted only when the service process exits cleanly. In this context, a clean exit means any of the following:

• exit code of 0;

• for types other than Type=oneshot, one of the signals SIGHUP, SIGINT, SIGTERM, or SIGPIPE;

• exit statuses and signals specified in SuccessExitStatus=.

If set to on-failure, the service will be restarted when the process exits with a non-zero exit code, is terminated by a signal (including on core dump, but excluding the aforementioned four signals), when an operation (such as service reload) times out, and when the configured watchdog timeout is triggered.

The restart can be tested by sending an appropriate signal to the application. See the manual at man kill for details.

Answer 3

You can call setsockopt(2) for listening socket with SO_REUSEADDR, so you will be able to bind(2) it again without waiting for expiring all connections. Another possibility: drop connections from kernel. FreeBSD have tcpdrop command for this, don't know about Linux.

Answer 4

If your linux distro uses Upstart instead of SysV init, then the respawn keyword does this.

http://upstart.ubuntu.com/cookbook/#respawn