24

I have deployed my application onto AWS EC2 and I want to implement automation where if I restart my instance or when the Nginx web server is down, it will restart by itself. I do not really know where to start with this.

I heard I can use crontab to schedule automatic monitoring and if it is down, it can send email alerts and restart the webserver.

7
  • What OS you will use in AWS? Feb 17, 2020 at 10:47
  • Hi @RomeoNinov, I am using linux Feb 17, 2020 at 11:01
  • Which distribution? Feb 17, 2020 at 11:02
  • 1
    RHEL, SuSE, Ubuntu, Debian... Feb 17, 2020 at 11:28
  • 8
    Why do you need the server to be restarted? What is crashing it? nginx is very stable anywhere I've used it. Just use the version you get from the package manager if you want it to integrate into your OS properly.
    – bobsburner
    Feb 18, 2020 at 9:06

8 Answers 8

11

Use monit which purpose is to take care of situations like this.

apt install monit

nano /etc/monit/conf.d/nginx.conf

Put content below inside this file and restart monit

check process nginx with pidfile /var/run/nginx.pid
start program = "/usr/sbin/service nginx start"
stop program = "/usr/sbin/service nginx stop"
12
  • Hi @iWizard, when I tried to save the file, it threw back an error saying no such file or directory. Do I have to mkdir the directory? Feb 17, 2020 at 12:32
  • @NewbeeCoding is your OS Ubuntu? If yes then monit should be under /etc/monit/. If iy there the just try to mkdir /etc/monit/conf.d
    – iWizard
    Feb 17, 2020 at 12:36
  • I am running, Windows 10 using Ubunto 18.04 LTS to connect to my application Feb 17, 2020 at 12:42
  • 1
    thanks for your solution. Got it working. Now when my Nginx is down, it starts again automatically. But I still can't access the UI to monitor it. but thanks for the help! Feb 17, 2020 at 14:27
  • 15
    This all feels like solving the wrong problem. Something else is wrong if this is even something you have to worry about, and even failing that, there's no reason to add complexity when systemd is already capable of restarting the service. Feb 18, 2020 at 21:16
89

It is a feature of SystemD. Override existing unit file for NGINX by running systemctl edit nginx then paste in:

[Service]
Restart=always

Save.

If NGINX is down due to, e.g. the OOM killer, it will be restarted after dying. If you have a configuration error in NGINX, it will not be restarted, of course.

To verify this configuration. start NGINX service with systemctl start nginx, and verify it is running with systemctl status nginx.

Kill it via pkill -f nginx. Confirm that NGINX is running anyway with systemctl status nginx.

8
  • Hi there, when I stop the service. It did not start again automatically, it says [Service] : command not found Feb 18, 2020 at 2:05
  • 1
    @NewbeeCoding That seems like you did not have a nginx.service file to start with. Is this nginx something you built yourself from source code?
    – Zan Lynx
    Feb 18, 2020 at 5:40
  • Hi, @ZanLynx, this was from a source code that had built it already Feb 18, 2020 at 5:51
  • 13
    @NewbeeCoding compiling/installing from source is not a good thing on a production system as it involves recompiling for upgrades and holding compilation software in prod is a security risk. Why not go for a packaged install and save a lot of trouble? If you use compiled install you need to set up at the minimum the startup script plus log rotation. For an example of a SystemD script look here. I recommend copying to /etc/systemd/system/nginx.service (/lib is where packaged SystemD unit files go). Feb 18, 2020 at 12:43
  • 1
    This should be the accepted answer
    – Climax
    Jan 18, 2022 at 5:56
5

Its actually fairly simple

Go to /lib/systemd/system Backup your ngnix systemd unit (in case) with sudo cp ngnix.service ngnix.service.old

Add the following 2 lines to the end of the service block in ngnix.service

Restart=on-failure
RestartSec=5s

Load the new config with sudo systemctl daemon-reload

To test - Lets try killing ngnix

cat /var/run/nginx.pid will give you the PID

sudo kill -9 PID will kill nginix

You'll find that if you check a the PID, there will be a different PID. If you hadn't run those lines, killing ngnix would result in a server that's down. This will only trigger a restart on a non graceful shutdown

2
  • 1
    /lib/systemd/system is reserved for unit files installed from packages. For custom units/overrides by admin there is a dedicated location, /etc/systemd/system. Feb 18, 2020 at 12:45
  • To expand on @DanilaVershinin note; it's possibly best to use sudo systemctl edit ngnix.service (or manually create /etc/systemd/system/nginx.service.d/override.conf) containing the desired extra config. Note that if you wish to override an existing value (that isn't boolean), you often need to set it to an empty value first, then set it to the desired value. If you wish to replace the default nginx.service completely, drop it in /etc/systemd/system/nginx.service. Feb 18, 2020 at 21:14
5

You already have lots of answers how to do it, but I'd investigate what is happening to make it shut down in the first place, and fix that.

When nginx crashes, all currently running requests will be terminated in an unknown state -- files half-transferred, API calls unreplied. In principle the application on top should deal with this, in practice they seldom do, but at that layer it will manifest as weird and unreproducible behaviour, giving the people using the service an feeling of instability (and rightfully so).

0

If you want to restart the process like you said in the question title, all the other answers seem great. If you want to restart the instance like you said in the question body, you can use an Auto Scaling Group.

Set up the Auto Scaling Group with minimum and maximum 1 instance, and point the healthcheck to the content served by your Nginx web server (ideally, something that doesn't consume a lot of CPU like robots.txt). The Auto Scaling Group will detect when your instance is unhealthy (might take 3 or 5 healthcheck fails) and will kill the instance and start a new one.

Keep in mind, you will lose all state stored in the instance (as the instance is actually destroyed). Keeping state inside a server is a cloud anti-pattern anyway. You'll also lose caches.

0

Tried to use systemctl edit nginx and pasted those lines:

RestartSec=5s

But getting this error and when nginx crashes its not restarted /etc/systemd/system/nginx.service.d/override.conf:1: Assignment outside of section. Ignoring.

2
0

auto restart on failures for Nginx on both the prod servers, staging server, and dev server. Server name as:

dev staging prod

I modified "/lib/systemd/system/nginx.service" and added the following line to [Service] section at the end -> Restart=on-failure

after changing the configuration we did, we have to execute the command ' systemctl daemon-reload ' for the auto-restart configuration to take effect.

This should restart the Nginx when it crashes

-3

If your distribution is based on systemd you can use sample script like this to start the daemon if it is not started/died

#!/bin/bash
systemctl is-active nginx
if [ "$?" -ne 0 ]
then systemctl start nginx
mail....... #send some mail
fi

the record in cron should be something like

*/5 * * * * /path/to/the/script

And this must be run as root (root cron)

2
  • 1
    This is definitely overcomplicating things. Why use an extra cron job to manage a systemd service? You're already using systemd, which is a service manager itself. Simply set the systemd unit file to restart the nginx service when it dies. Feb 20, 2020 at 10:21
  • @MartijnHeemels, I create it based on my knowledge of systemd. If systemd can do it via configuration of the server OK, but I was unaware of this fact. Should I repeat what other answer? Are negative votes indicator of wrong or harm solution? Feb 20, 2020 at 10:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .