30

I need to have network messages sent when a systemd service I have crashes or is hung (i.e., enters failed state; I monitor for hung by using WatchdogSec=). I noticed that newer systemd have FailureAction=, but then saw that this doesn't allow arbitrary commands, but just rebooting/shutdown.

Specifically, I need a way to have one network message sent when systemd detects the program has crashed, and another when it detects it has hung.

I'm hoping for a better answer than "parse the logs", and I need something that has a near-instant response time, so I don't think a polling approach is good; it should be something triggered by the event occurring.

  • Can you please provide an example (even a non working one) of what you are trying to achieve? – dawud May 27 '15 at 19:05
  • Can you please edit your question to add this information instead of adding it in a comment? :) – dawud May 27 '15 at 19:11
31

systemd units support OnFailure that will activate a unit (or more) when the unit goes to failed. You can put something like

 OnFailure=notify-failed@%n

And then create the notify-failed@.service service where you can use the required specifier (you probably will want at least %i) to launch the script or command that will send notification.

You can see a practical example in http://northernlightlabs.se/systemd.status.mail.on.unit.failure

  • 5
    There are a couple corrections needed to the instructions on the linked site. First, notify%n.service is redundant, and will result in notify@my-service.service.service. Second, %i should be used instead of %I, or all dashes in the name will be converted to forward slashes. – orodbhen Jun 22 '16 at 15:42
  • 3
    Is there a way to do this for multiple or all units, without modifying their unit files? – Vladimir Panteleev Sep 10 '17 at 12:52
12

Just my way to notify :

/etc/systemd/system/notify-email@.service

[Unit]
Description=Sent email 

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c '/usr/bin/systemctl status %i | /usr/bin/mailx -Ssendwait -s "[SYSTEMD_%i] Fail" your_admin@company.blablabla'

add to systemd :

systemctl enable /etc/systemd/system/notify-email@service

At others services add :

[Unit]
OnFailure=notify-email@%i.service

Reload de configurations :

systemctl daemon-reload
0

I came across this utility which seems to provide this: https://github.com/joonty/systemd_mon

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.