Logstash Forwarder doesn't start up with chkconfig in CentOS 5

Question

I have set up logstash-forwarder on a CentOS 5 machine installing it from this RPM:

http://download.elasticsearch.org/logstash-forwarder/packages/logstash-forwarder-0.3.1-1.x86_64.rpm

When I went through the process I found the init script provided for CentOS/RHEL/Fedora didn't work as expected. I got no information fed back to me when running any commands and it didn't start the process. I found someone else had some similar problems and had written their own, this script works fine when I start up logstash-forwarder using service or the init.d command, however, I have noticed that when I reboot the server the service doesn't start up (the following is from just after a reboot):

# service logstash-forwarder status
logstash-forwarder dead but pid file exists

# chkconfig --list logstash-forwarder
logstash-forwarder      0:off   1:off   2:off   3:on    4:on    5:on    6:off

If I then start it manually, it all works fine. Some research has told me that I should have a file here (as I use run level 3):

# ls -la /etc/rc3.d/ | grep logstash
lrwxrwxrwx  1 root root   28 Nov  5 12:33 S99logstash-forwarder -> ../init.d/logstash-forwarder

Which is in place and is only followed by smartd in the numerical list, so I don't think the issue is that services that logstash-forwarder relies on are not fired up already. Can anyone advise how further to investigate in to why this isn't starting up?

EDIT

I have noticed this issue is not apparent on a CentOS 6 system, only on a CentOS 5 machine. I am not sure what difference this makes as the init script template files are essentially the same. Anyone?

FURTHER EDIT

I have had to modify the init script slightly, this was just down to the fact I was shipping /var/log/messages to my logstash server, but the init script made the logstash logs go in to messages, causing a feedback loop, every time it sent a log to logstash it would add a log to messages, which would then be sent to logstash. So I have changed the init script slightly:

#! /bin/sh
#
# chkconfig: 345 99 99
# description: logstash-forwarder
# processname: logstash-forwarder
# config: /etc/logstash-forwarder

# Source function library.
. /etc/init.d/functions

NAME=logstash-forwarder
DESC="log shipper"
PATH=/sbin:/usr/sbin:/bin:/usr/bin
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
DAEMON=/opt/logstash-forwarder/bin/logstash-forwarder
DAEMON_ARGS="-config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100"

start() {
        echo -n "Starting $NAME: "
        daemon --pidfile $PIDFILE "nohup $DAEMON $DAEMON_ARGS >/dev/null 2>&1 &"
        touch /var/lock/subsys/$NAME
        echo $(pidofproc $NAME) > $PIDFILE
        echo
        return 0
}

stop() {
        echo -n "Shutting down $NAME: "
        killproc -p $PIDFILE
        rm -f /var/lock/subsys/$NAME
        echo
        return 0
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    status)
        status -p $PIDFILE $NAME
        ;;
    restart)
        stop
        start
        ;;
    *)
        echo "Usage: service $NAME {start|stop|status|restart}"
        exit 1
        ;;
esac
exit $?

This script works perfectly well on CentOS 6, but when I reboot in CentOS 5 I have the issue explained above, any help would be greatly appreciated.

ANOTHER EDIT

I have done some further testing, and found that when I reboot the machine logstash-forwarder does start up, but the logstash-forwarder.pid file is empty, so I guess the init script is saying it's dead just because it cannot find the process as it's not noted in the pid file. The following is from straight after a reboot:

# service logstash-forwarder status
logstash-forwarder dead but pid file exists
# ps -ef | grep logstash
root      3297     1  2 10:20 ?        00:00:02 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100 -log-to-syslog
root      3722  3324  0 10:22 pts/0    00:00:00 grep logstash
# cat /var/run/logstash-forwarder.pid

# service logstash-forwarder restart
Shutting down logstash-forwarder:                          [FAILED]
Starting logstash-forwarder:                               [  OK  ]
# cat /var/run/logstash-forwarder.pid
3748 3297
# ps -ef | grep logstash
root      3297     1  2 10:20 ?        00:00:03 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100 -log-to-syslog
root      3748     1  1 10:22 pts/0    00:00:00 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100 -log-to-syslog
root      3767  3324  0 10:22 pts/0    00:00:00 grep logstash
# service logstash-forwarder restart
Shutting down logstash-forwarder:                          [  OK  ]
Starting logstash-forwarder:                               [  OK  ]
# ps -ef | grep logstash
root      3788     1  5 10:22 pts/0    00:00:00 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100 -log-to-syslog
root      3796  3324  0 10:22 pts/0    00:00:00 grep logstash
# cat /var/run/logstash-forwarder.pid
3788

So it looks like the service starts, no entry is made in the pid file, restart the process I get 2 copies of the service running, as it doesn't kill the first process, but the second time I restart it both processes are killed and I am back to having one process.

Now I need to figure out why the pid file entry isn't altered when I boot the machine.

Answer 1

I ended up posting about the issue on the github page for the project, and I got quite a speedy response pointing me to another issue which came along with another init script, which I have now implemented. This seems to work correctly.

Thanks to TrevorH in the CentOS IRC channel for the assitance, as well as driskell on github for getting me to the new init script, I'll copy it below for reference in case anyone has the same problem I have with that script above:

#!/bin/bash
# chkconfig: 345 80 20
# description: Logstash Forwarder
# processname: logstash-forwarder
# config: /etc/logstash-forwarder
# pidfile: /var/run/logstash-forwarder.pid

### BEGIN INIT INFO
# Provides: logstash-forwarder
# Required-Start: $local_fs $network $remote_fs
# Required-Stop: $local_fs $network $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: start and stop logstash-forwarder
# Description: Logstash Forwarder
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

PATH=/sbin:/usr/sbin:/bin:/usr/bin

prog=logstash-forwarder
DAEMON=/opt/logstash-forwarder/bin/logstash-forwarder
pidfile=/var/run/$prog.pid
lockfile=/var/lock/subsys/$prog

# load defaults

[ -e /etc/default/$prog ] && . /etc/default/$prog
[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

DAEMON_ARGS="${DAEMON_ARGS:--config /etc/logstash-forwarder -spool-size 100 -log-to-syslog}"

start()
{
    echo -n $"Starting $prog: "
    nohup $DAEMON $DAEMON_ARGS </dev/null >/dev/null 2>&1 &
    retval=$?
    pid=$!
    echo $pid > $pidfile
    if [ rh_status_q ]; then
        touch $lockfile
        success
        echo
    fi
    return $retval
}

stop()
{
    echo -n $"Stopping $prog: "
    killproc -p "$pidfile" $prog
    retval=$?
    [ -f "$pidfile" ] && rm -f $pidfile
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    stop
    start
}

reload() {
    restart
}

force_reload() {
    restart
}

rh_status() {
    status -p $pidfile $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
        ;;
    status)
        rh_status
        ;;
    *)
        echo "Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
exit $?

This has been modified slightly as the original script had .conf in the DAEMON_ARGS statement which wasn't needed on CentOS 5 or 6

EDIT

There were some issues with the init script, but the commiter has updated it today, for my instance I also needed either of the following files:

/etc/defaults/logstash-forwarder
/etc/sysconfig/logstash-forwarder

Content:

DAEMON_ARGS="${DAEMON_ARGS:--config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100}" 

Answer 2

Sounds like logstash-forwarder is not properly being stopped/killed during the shutdown process.

Have you manually tried to logstash-forwarder using the init script, and see how long it actually takes for the daemon to be fully stoped? Are the logs logstash-forwarder is tracking in your CentOS 5 machine, large log files compared to what your CentOS 6 machine is tracking?

As a last resort you can pass a SIGKILL to killproc to ensure logstash-forwarder stops before the reboot.

killproc -p $PIDFILE -9