19

I have a critical application which is run as a service by systemd.

It is set up to restart as soon as there is a failure.

How to send an email if the application restarts?

1
  • 2
    I don't know how to do it with systemd, but with monit you can have it watch a process, and it can send out a notification if the process id changes.
    – Zoredache
    Sep 30, 2017 at 2:55

4 Answers 4

33

First you need two files: an executable for sending the mail and a .service for starting the executable. For this example, the executable is just a shell script using sendmail:

/usr/local/bin/systemd-email:

#!/bin/bash

/usr/bin/sendmail -t <<ERRMAIL
To: $1
From: systemd <root@$HOSTNAME>
Subject: $2
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

$(systemctl status --full "$2")
ERRMAIL

Whatever executable you use, it should probably take at least two arguments as this shell script does: the address to send to and the unit file to get the status of. The .service we create will pass these arguments:

/etc/systemd/system/[email protected]:

[Unit]
Description=status email for %i to user

[Service]
Type=oneshot
ExecStart=/usr/local/bin/systemd-email address %i
User=nobody
Group=systemd-journal

Where user is the user being emailed and address is that user's email address. Although the recipient is hard-coded, the unit file to report on is passed as an instance parameter, so this one service can send email for many other units. At this point you can start [email protected] to verify that you can receive the emails.

Then simply edit the service you want emails for and add OnFailure=status-email-user@%n.service to the [Unit] section. %n passes the unit's name to the template.

Source: archlinux wiki: systemd timers MAILTO

9
  • But @Dave doesn't require another service. He want the service he already uses to be able to send mail on each start/restart. For that, there is ExecStartPost option. Sep 30, 2017 at 12:57
  • @JaroslavKucera I think this to decide it up to the OP... :) Also, I'm not sure if ExecStartPost is the right choice: It would also trigger after a "normal" start, not only in case of a failure, right?
    – gxx
    Sep 30, 2017 at 13:14
  • I'm interested why this get's down voted - please raise your voices and speak up!
    – gxx
    Sep 30, 2017 at 13:15
  • Because the question isn't about another service, but modification of existing service. Yes, ExecStartPost would trigger mail send even on normal start. I'm not aware of anything which would work only on restart. Sep 30, 2017 at 13:17
  • @JaroslavKucera Well, it seems the OP disagrees, but of course, keep your opinion.
    – gxx
    Sep 30, 2017 at 13:18
14

The solution proposed by @gf_ worked well for our situation running clickhouse on CentOS7. Clickhouse crashes somewhat regularly on us so we needed to have it both restarted automatically and be notified when the restart occurred. While it seems a little clunky to add a second service to systemd, this is necessary due to systemd's design.

That being said, this solution, when combined with auto-restarting, stopped working for us when we deployed to CentOS8. This is because systemd v239 shipped in C8 introduced a change to the OnFailure= semantics when combined with a non-default configuration of Restart= (Restart=on-failure in our case). The new OnFailure= behavior only triggers the one-shot service if the restart failed completely, not just after a crash. This newer behavior would happily restart the service, but we would not get the email as OnFailure= was no longer being invoked.

Note our primary expectation: we wanted systemd to restart the process AND send an email notification. The v239 update made our previous solution cited by gf_ not work anymore. Fortunately we were able to get this working.

Our solution is to use ExecStopPost to invoke the email notification script. This works fine, but now a new issue came up: an email notification was sent when the clickhouse service started normally, such as on server startup. While not a big deal, ideally we wanted to get email notifications only on crashes. We were able to achieve this by adding the following code to our email script:

# Don't do anything if the service intentionally stopped successfully. if [ $SERVICE_RESULT == "success" ]; then exit fi

... $SERVICE_RESULT is an environment variable supplied by systemd to the target process of ExecStopPost. By checking for a success result, we assume that this invocation came from a normal startup, or shutdown, and do nothing. On any other value, such as signal, the script would continue on an send an email. The possible values of this variable are stated in the documentation.

Thanks to gf_ for the initial solution. I hope people find my update helpful for CentOS8. Some more links that helped me out:

  1. https://superuser.com/questions/1360346/how-to-send-an-email-alert-when-a-linux-service-has-stopped
  2. https://unix.stackexchange.com/questions/422933/confusing-systemd-behaviour-with-onfailure-and-restart
  3. https://unix.stackexchange.com/questions/197636/run-an-arbitrary-command-when-a-service-fails
1

You may try to use systemd service option ExecStartPost.

The description is available here:

https://www.freedesktop.org/software/systemd/man/systemd.service.html

There can be more declarations of this option in the service definition file. It's triggered one by one.

You'll have some examples in your system as well.

-1

You can create a shell script to check service status and send email while server booting up. This link might help you

https://askubuntu.com/questions/814/how-to-run-scripts-on-start-up

1
  • 1
    This question is not about booting, but about a service restarting, which might happen quite some time after booting. Hence, I'm unsure if your answer is of any help.
    – gxx
    Sep 30, 2017 at 7:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .