Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to restart Apache and Varnish and email some logs when the script can't fetch robots.txt but I am getting an error

./healthcheck: 43 [[: not found

My server is Ubuntu 12.04 64-bit

#!/bin/sh
# Check if can fetch robots.txt if not then restart Apache and Varnish
# Send last few lines of logs with date via email

PATH=/bin:/usr/bin
THEDIR=/tmp/web-server-health
EMAIL=dude@email.net
mkdir -p $THEDIR

if ( wget --timeout=30 -q -P $THEDIR http://website.com/robots.txt )
then
    # we are up
    touch ~/.apache-was-up
else
    # down! but if it was down already, don't keep spamming
    if [[ -f ~/.apache-was-up ]]
    then
        # write a nice e-mail
        echo -n "Web server down at " > $THEDIR/mail
        date >> $THEDIR/mail
        echo >> $THEDIR/mail
        echo "Apache Log:" >> $THEDIR/mail
        tail -n 30 /var/log/apache2/error.log >> $THEDIR/mail
        echo >> $THEDIR/mail
        echo "AUTH Log:" >> $THEDIR/mail
        tail -n 30 /var/log/auth.log >> $THEDIR/mail
        echo >> $THEDIR/mail
        # kick apache
        echo "Now kicking apache..." >> $THEDIR/mail
        /etc/init.d/varnish stop >> $THEDIR/mail 2>&1
        killall -9 varnishd >> $THEDIR/mail 2>&1
        /etc/init.d/varnish start >> $THEDIR/mail 2>&1
        /etc/init.d/apache2 stop >> $THEDIR/mail 2>&1
        killall -9 apache2 >> $THEDIR/mail 2>&1
        /etc/init.d/apache2 start >> $THEDIR/mail 2>&1
        # prepare the mail
        echo >> $THEDIR/mail
        echo "Good luck troubleshooting!" >> $THEDIR/mail
        # send the mail
        sendemail -o message-content-type=html -f alerts@website.net -t $EMAIL -u ALARM -m < $THEDIR/mail
        rm ~/.apache-was-up
    fi
fi

rm -rf $THEDIR
share|improve this question
if [[ -f ~/.apache-was-up ]]

In Ubuntu, /bin/sh is a symlink to dash instead of bash. dash is a minimal shell designed for scripts, whereas bash is more featured and designed for interactive use. The [[ <expression> ]] syntax is only implemented in bash, not in dash, and therefore you receive that error. The [ <expression> ] syntax is enough for what you need though, so change that line to this:

if [ -f ~/.apache-was-up ]
share|improve this answer
    
After copying Jay's script I changed the top line to /bin/bash and it worked fine the first three times I ran it then it just stopped working. – Tyler Jun 12 '12 at 2:13

I think this is what you're looking for (I prefer /tmp since it's removed after a reboot, so a fresh start).

#!/bin/sh
# Check if can fetch robots.txt if not then restart Apache and Varnish
# Send last few lines of logs with date via email

PATH=/bin:/usr/bin
THEDIR=/tmp/web-server-health
EMAIL=dude@email.net
mkdir -p $THEDIR

if ( wget --timeout=30 -q -P $THEDIR http://website.com/robots.txt )
then
    # we are up
    touch /tmp/.apache-was-up
else
    # down! but if it was down already, don't keep spamming
    if [  -f /tmp/.apache-was-up ];
    then
        # write a nice e-mail
        echo -n "Web server down at " > $THEDIR/mail
        date >> $THEDIR/mail
        echo >> $THEDIR/mail
        echo "Apache Log:" >> $THEDIR/mail
        tail -n 30 /var/log/apache2/error.log >> $THEDIR/mail
        echo >> $THEDIR/mail
        echo "AUTH Log:" >> $THEDIR/mail
        tail -n 30 /var/log/auth.log >> $THEDIR/mail
        echo >> $THEDIR/mail
        # kick apache
        echo "Now kicking apache..." >> $THEDIR/mail
        /etc/init.d/varnish stop >> $THEDIR/mail 2>&1
        killall -9 varnishd >> $THEDIR/mail 2>&1
        /etc/init.d/varnish start >> $THEDIR/mail 2>&1
        /etc/init.d/apache2 stop >> $THEDIR/mail 2>&1
        killall -9 apache2 >> $THEDIR/mail 2>&1
        /etc/init.d/apache2 start >> $THEDIR/mail 2>&1
        # prepare the mail
        echo >> $THEDIR/mail
        echo "Good luck troubleshooting!" >> $THEDIR/mail
        # send the mail
        sendemail -o message-content-type=html -f alerts@website.net -t $EMAIL -u ALARM -m < $THEDIR/mail
        rm /tmp/.apache-was-up
    fi
fi

rm -rf $THEDIR
share|improve this answer
    
This works perfectly thank you Jay! – Tyler Jun 12 '12 at 1:24
    
Hmmm very strange, it worked 3 times and now it doesn't work anymore. – Tyler Jun 12 '12 at 1:43
    
@Tyler Doesn't work in what way? – mgorven Jun 12 '12 at 4:20
    
I forgot to change one of the ~/.apache-was-up's. How about now? – Jay Jun 12 '12 at 10:44

While the script may work, there are some issues with it, like unsafe use of temporary files.

You really should have a look at Monit, its open source and it is easy to write a rule that will restart your web services based on service responses. It can also detect if a running process dies (e.g. its PID goes away) and you have the possibility to temporary deactivate a check (e.g. when you are doing a service maintenance).

share|improve this answer
    
I added some instructions below on how to use Monit specifically in this use case. – Evan Donovan Feb 25 at 16:36

Here's a detailed answer on how to use Monit to monitor and restart Varnish if it is down.

First, visit http://www.tecmint.com/how-to-install-and-setup-monit-linux-process-and-services-monitoring-program/ to get some basic information on Monit and get it installed. This is a helpful tutorial since it gives the default paths for both RHEL/CentOS and Debian/Ubuntu.

Second, visit https://www.computerminds.co.uk/articles/monitoring-varnish and follow the instructions about setting up Varnish monitoring.

Here are a few notes from my setup:

  • I would recommend using the URL /varnish-ping for the VCL ping URL. It's a logical path that is easy to remember. You can test this out once set up by running curl 127.0.0.1:80/varnish-ping from the command line & making sure it returns a 200 response.
  • Make sure you have the right Varnish pid. As stated in the second article, you can find that in the init script. Mine was actually /var/run/varnish.pid. You can do an ls to make sure that file exists while Varnish is running.
  • You will probably want to allow access to the Monit web interface to help in monitoring. If you have a firewall, you will first need to add port 2812 to the allowed ports. Then, change which IP's are allowed to access as per http://digitizor.com/how-to-enable-monit-web-interface-from-all-ips-and-change-port/. You can specify your own remote IP instead of 0.0.0.0/8 for greater security. Note that the password is passed through HTTP Auth, so you should not use a password that you want to keep particularly secure.
  • Make sure that the Monit log is active. Check the "set log" log to make sure it has something like set log /var/log/monit.log on it.
  • If installing Varnish doesn't put it on the list of daemons to start on a system reboot, you will have to add it via chkconfig (or the Ubuntu/Debian equivalent). I ran chkconfig --list to confirm it was present.

Hope that helps present a more reliable alternative. Note that you can also add in email monitoring for services via Monit as well. Here's some more details specifically about email alerts: https://fak3r.com/2010/04/10/howto-use-monit-to-monitor-sites-and-alert-users/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.