Reboot if tomcat7 service is not responding

Question

I am running a web app on a Tomcat server. There is a hard-to-detect problem within the server code that causes it to crash once or twice everyday. I will dig in to correct it when I have time. But until that day, in a problematic case restarting tomcat (/etc/init.d/tomcat7 restart) or basically rebooting the machine also seem pretty good solutions for now. I want to detect liveliness of server with wget instead of grep or something else because even though tomcat is running my service my be down.

wget localhost:8080/MyService/

outputs

--2012-12-04 14:10:20--  http://localhost:8080/MyService/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2777 (2.7K) [text/html]
Saving to: “index.html.3”

100%[======================================>] 2,777       --.-K/s   in 0s

2012-12-04 14:10:20 (223 MB/s) - “index.html.3” saved [2777/2777]

when my service is up. And outputs

Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8080... failed: Connection refused.

or just stucks after saying

--2012-12-04 14:07:34--  http://localhost:8080/MyService/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response...

Can you offer me a shell script with a cron job or something else to do that. I prefer not to use cron if there is an alternative.

Answer 1

I hope you have already found to root cause for your problem and been able to fix it properly. In case you or someone else would need a solution for this, here is a try for an answer.

The thing here is that your service may sometimes 'hang', and the monitoring must also be able to catch it up. In the simple script below we place the wget status query to background, wait a few seconds and if it has not been able to retrieve status 200 from the service, restart it.

#!/bin/sh
# WARNING, UNTESTED CODE !

TMPFILE=`mktemp`
WAITTIME=15

# Run the test
wget localhost:8080/MyService/ -o $TMPFILE &
WGETPID=$!

# Wait few seconds and let the test finish
sleep $WAITTIME

if [ ! `grep "HTTP request sent" $TMPFILE |grep "200 OK"|wc -l` -gt 0 ]; then
    echo "The service did not return 200 in $WAITTIME seconds."
    echo "Restarting it."
    /etc/init.d/tomcat7 restart
fi

# Cleanup
rm $TMPFILE
kill $WGETPID

For scheduling, I really recommend cron for simplicity. Another choice would be to start this as a daemon, which would introduce unnecessary complexity, IMHO. Also some other (external) scheduler could be used, but I keep the cron simplest.

Hopefully this helps.