Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need to monitor several processes running on my webserver. For some reason, varnish currently crashes once every day or two. I'm using monit to supposedly restart varnish automatically, but it doesn't work. Here's my monit.conf entry for Varnish.

check process varnish with pidfile /var/run/varnish.pid
    start program = "/etc/init.d/varnish start" with timeout 60 seconds
    stop program = "/etc/init.d/varnish stop"
    if failed host <my server ip> port 80 protocol http
        and request "/blank.html" then restart
    if 3 restarts within 5 cycles then timeout
    group server

The log file shows that after varnish stops running, the attempted restarts afterwards all fail. Then eventually monit stops monitoring varnish.

Anyone have suggestions for how I can fix this? Or better yet, can you suggest other simple ways of automatically monitoring and restarting crashed processes? Thanks!

share|improve this question

4 Answers 4

up vote 13 down vote accepted

I'd look in to daemontools (http://cr.yp.to/daemontools.html).

Supervise was built for exactly this purpose -- to start processes and watch them, restarting them immediately if they ever terminate.

You could still use monit if you need to do anything more complicated than a simple "is it still running" check, and if the process needs to be restarted, then do that through supervise.

share|improve this answer
    
I use the daemontools too, for monitoring unstable services processes. Quite handy if i had to say. :-) –  edomaur Aug 11 '09 at 22:59
    
That's very useful. Thanks! –  Lin Aug 12 '09 at 2:21

You could also use /etc/inittab to restart dead processes using the respawn action.

See inittab section on http://aplawrence.com/Unixart/startup.html

share|improve this answer

You can use event handler scripts with Nagios if you have that in place to restart services.

If varnish requires root permission to start (init.d scripts usually do) change "/etc/init.d/varnish start" to "sudo /etc/init.d/varnish start". But that probably won't be quite enough since you probably don't want to give whatever user monit runs as total sudo nopasswd privileges to all commands and giving sudo to a shell script would be basically just as bad. So you are going to need to figure out which commands in that init script need sudo, give those commands sudo privileges in the /etc/sudoers file to the monit user, and the finally edit that init script accordingly. Or maybe instead of all this varnish can be run as non-root user?

Finally, I am sure you know this but I am going to say it anyways. You are clearly putting a lot of effort into this, I hope you are putting as much effort into figuring out why varnish is crashing and actually fixing it (or hounding the developers to figure out why) :-)

Update:
This might not be as clean, but an easy way to get this done as root might be to set up a script that checks if the process is okay, and if not starts it. Then just run that script every couple minutes as a cron job.

share|improve this answer
    
I considered Nagios at first, but wanted something small and simple for my purposes. And yes, I'm looking into the Varnish issue. One of my servers has been running it stable for a very long time, so it's definitely got to do with me. :( –  Lin Aug 12 '09 at 2:23

Another great method taken from StackOverflow:

until myserver; do
    echo "Server 'myserver' crashed with exit code $?.  Respawning.." >&2
    sleep 1
done

This could be added to the crontab:

crontab -e

Then add a rule to start your monitor script:

@reboot /usr/local/bin/myservermonitor

Or added as a script in /etc/init.d

See the StackOverflow answer for a detailed explanation on why this is a good approach.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.