13

I have deployed my application onto AWS EC2 and I want to implement automation where if I restart my instance or when the Nginx web server is down, it will restart by itself. I do not really know where to start with this.

I heard I can use crontab to schedule automatic monitoring and if it is down, it can send email alerts and restart the webserver.

  • What OS you will use in AWS? – Romeo Ninov Feb 17 at 10:47
  • Hi @RomeoNinov, I am using linux – Newbee Coding Feb 17 at 11:01
  • Which distribution? – Romeo Ninov Feb 17 at 11:02
  • 1
    RHEL, SuSE, Ubuntu, Debian... – Romeo Ninov Feb 17 at 11:28
  • 8
    Why do you need the server to be restarted? What is crashing it? nginx is very stable anywhere I've used it. Just use the version you get from the package manager if you want it to integrate into your OS properly. – bobsburner Feb 18 at 9:06
6

Use monit which purpose is to take care of situations like this.

apt install monit

nano /etc/monit/conf.d/nginx.conf

Put content below inside this file and restart monit

check process nginx with pidfile /var/run/nginx.pid
start program = "/usr/sbin/service nginx start"
stop program = "/usr/sbin/service nginx stop"
|improve this answer|||||
  • Hi @iWizard, when I tried to save the file, it threw back an error saying no such file or directory. Do I have to mkdir the directory? – Newbee Coding Feb 17 at 12:32
  • @NewbeeCoding is your OS Ubuntu? If yes then monit should be under /etc/monit/. If iy there the just try to mkdir /etc/monit/conf.d – iWizard Feb 17 at 12:36
  • I am running, Windows 10 using Ubunto 18.04 LTS to connect to my application – Newbee Coding Feb 17 at 12:42
  • 1
    thanks for your solution. Got it working. Now when my Nginx is down, it starts again automatically. But I still can't access the UI to monitor it. but thanks for the help! – Newbee Coding Feb 17 at 14:27
  • 13
    This all feels like solving the wrong problem. Something else is wrong if this is even something you have to worry about, and even failing that, there's no reason to add complexity when systemd is already capable of restarting the service. – Michael - sqlbot Feb 18 at 21:16
58

It is a feature of SystemD. Override existing unit file for NGINX by running systemctl edit nginx then paste in:

[Service]
Restart=always

Save.

If NGINX is down due to, e.g. the OOM killer, it will be restarted after dying. If you have a configuration error in NGINX, it will not be restarted, of course.

To verify this configuration. start NGINX service with systemctl start nginx, and verify it is running with systemctl status nginx.

Kill it via pkill -f nginx. Confirm that NGINX is running anyway with systemctl status nginx.

|improve this answer|||||
  • Hi there, when I stop the service. It did not start again automatically, it says [Service] : command not found – Newbee Coding Feb 18 at 2:05
  • 1
    @NewbeeCoding That seems like you did not have a nginx.service file to start with. Is this nginx something you built yourself from source code? – Zan Lynx Feb 18 at 5:40
  • Hi, @ZanLynx, this was from a source code that had built it already – Newbee Coding Feb 18 at 5:51
  • 12
    @NewbeeCoding compiling/installing from source is not a good thing on a production system as it involves recompiling for upgrades and holding compilation software in prod is a security risk. Why not go for a packaged install and save a lot of trouble? If you use compiled install you need to set up at the minimum the startup script plus log rotation. For an example of a SystemD script look here. I recommend copying to /etc/systemd/system/nginx.service (/lib is where packaged SystemD unit files go). – Danila Vershinin Feb 18 at 12:43
  • @NewbeeCoding I agree 100% with Danila that you should avoid compiling your own software for production systems, if at all possible. I wrote an elaborate answer on that very topic that you might find informative (the answer is about Apache on Debian, but the principle is exactly the same). – marcelm Feb 20 at 15:17
5

You already have lots of answers how to do it, but I'd investigate what is happening to make it shut down in the first place, and fix that.

When nginx crashes, all currently running requests will be terminated in an unknown state -- files half-transferred, API calls unreplied. In principle the application on top should deal with this, in practice they seldom do, but at that layer it will manifest as weird and unreproducible behaviour, giving the people using the service an feeling of instability (and rightfully so).

|improve this answer|||||
0

Its actually fairly simple

Go to /lib/systemd/system Backup your ngnix systemd unit (in case) with sudo cp ngnix.service ngnix.service.old

Add the following 2 lines to the end of the service block in ngnix.service

Restart=on-failure
RestartSec=5s

Load the new config with sudo systemctl daemon-reload

To test - Lets try killing ngnix

cat /var/run/nginx.pid will give you the PID

sudo kill -9 PID will kill nginix

You'll find that if you check a the PID, there will be a different PID. If you hadn't run those lines, killing ngnix would result in a server that's down. This will only trigger a restart on a non graceful shutdown

|improve this answer|||||
  • 1
    /lib/systemd/system is reserved for unit files installed from packages. For custom units/overrides by admin there is a dedicated location, /etc/systemd/system. – Danila Vershinin Feb 18 at 12:45
  • To expand on @DanilaVershinin note; it's possibly best to use sudo systemctl edit ngnix.service (or manually create /etc/systemd/system/nginx.service.d/override.conf) containing the desired extra config. Note that if you wish to override an existing value (that isn't boolean), you often need to set it to an empty value first, then set it to the desired value. If you wish to replace the default nginx.service completely, drop it in /etc/systemd/system/nginx.service. – Jeremy Davis Feb 18 at 21:14
0

If you want to restart the process like you said in the question title, all the other answers seem great. If you want to restart the instance like you said in the question body, you can use an Auto Scaling Group.

Set up the Auto Scaling Group with minimum and maximum 1 instance, and point the healthcheck to the content served by your Nginx web server (ideally, something that doesn't consume a lot of CPU like robots.txt). The Auto Scaling Group will detect when your instance is unhealthy (might take 3 or 5 healthcheck fails) and will kill the instance and start a new one.

Keep in mind, you will lose all state stored in the instance (as the instance is actually destroyed). Keeping state inside a server is a cloud anti-pattern anyway. You'll also lose caches.

|improve this answer|||||
-3

If your distribution is based on systemd you can use sample script like this to start the daemon if it is not started/died

#!/bin/bash
systemctl is-active nginx
if [ "$?" -ne 0 ]
then systemctl start nginx
mail....... #send some mail
fi

the record in cron should be something like

*/5 * * * * /path/to/the/script

And this must be run as root (root cron)

|improve this answer|||||
  • 1
    This is definitely overcomplicating things. Why use an extra cron job to manage a systemd service? You're already using systemd, which is a service manager itself. Simply set the systemd unit file to restart the nginx service when it dies. – Martijn Heemels Feb 20 at 10:21
  • @MartijnHeemels, I create it based on my knowledge of systemd. If systemd can do it via configuration of the server OK, but I was unaware of this fact. Should I repeat what other answer? Are negative votes indicator of wrong or harm solution? – Romeo Ninov Feb 20 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.