0

I've installed logstash on my headless server but I'm having a few problems getting it to work from the upstart script with sudo service logstash start. This is the output I get from /var/log/logstash/logstash.log is here but I can't see anything obvious in there.

with the upstart script being (/etc/init/logstash.conf) # logstash - agent instance #

description     "logstash agent"

start on virtual-filesystems
stop on runlevel [06]

# Respawn it if the process exits
respawn

# We're setting high here, we'll re-limit below.
limit nofile 65550 65550

setuid logstash
setgid logstash

# You need to chdir somewhere writable because logstash needs to unpack a few
# temporary files on startup.
console log
script
  # Defaults
  PATH=/bin:/usr/bin
  LS_HOME=/var/lib/logstash
  LS_HEAP_SIZE="500m"
  LS_JAVA_OPTS="-Djava.io.tmpdir=${LS_HOME}"
  LS_LOG_FILE=/var/log/logstash/logstash.log
  LS_USE_GC_LOGGING=""
  LS_CONF_DIR=/etc/logstash/conf.d
  LS_OPEN_FILES=16384
  LS_NICE=19
  LS_OPTS="--debug"

  # Override our defaults with user defaults:
  [ -f /etc/default/logstash ] && . /etc/default/logstash

  HOME="${HOME:-$LS_HOME}"
  JAVA_OPTS="${LS_JAVA_OPTS}"
  # Reset filehandle limit
  ulimit -n ${LS_OPEN_FILES}
  cd "${LS_HOME}"

  # Export variables
  export PATH HOME JAVA_OPTS LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING
  test -n "${JAVACMD}" && export JAVACMD

  exec nice -n ${LS_NICE} /opt/logstash/bin/logstash agent -f "${LS_CONF_DIR}" -l "${LS_LOG_FILE}" ${LS_OPTS}
end script

With the logstash conf file being:

input {
  file {
    path => "/var/log/apache2/access.log"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
  match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    host => localhost
  }
  stdout { codec => rubydebug }
}

Nothing seems to even though there is definitely stuff happening in the apache log.

However, when I take the interesting stuff out of the upstart script (the stuff inbetween the script and end script and run it as a shell script, it works perfectly and I can see stuff coming into the logs. Can anyone give me some pointers on why this is happening? I'm running Ubuntu server 14.04 server and installed logstash=1.4.2-1-2c0f5a1

4
  • Any messages in /var/log/message and /var/log/logstash/ ? Have you tried testing your config with ${path_to}/logstash -t -f ${path_to_config} ?
    – stimur
    Feb 26 '15 at 14:07
  • The log output from `/var/log/logstash/logstash.conf is here: pastebin.com/AcdiUSM6. The config file works fine as I am able to run it from the command line, just not when run as a service.
    – Sonoman
    Feb 27 '15 at 16:52
  • and you probably run tests as root and run service as logstash ? :) try testing from logstash user. try clearing log file before next run to see what is recent output.
    – stimur
    Feb 27 '15 at 17:56
  • You were absolutely right! I was running the test as root and the service as logstash user. It definitely looks like a user issue but I still can't figure it out. The debug logs from the test run as root which works are here (pastebin.com/pzc634Xt) and the logs from the service (run as logstash which doesn't work are here: pastebin.com/CKp7Lwpq). Can you offer any more advice?
    – Sonoman
    Feb 28 '15 at 14:06
1

So I suppose this can be an answer now. Looks like your tests were run as root user. and your service is running as logstash. Your logs from tests do not show any errors. But I think your problem is with file permissions. Please check that config files are readable for user logstash and log files are writable for him. Plus, since that process should be able to read apache log files directly, check if logstash is to read them as well. If logstash is configured to listen for incoming network connections, check of ports configured are higher than 1024.

Also you can try running your tests as logstash user:

something like that:

su -c "/path/to/logstash -t -f /path/to/config" logstash
1
  • Thank you! The issue was down to the apache log not being readable. Can't believe I've been stuck for so long with something so obvious!
    – Sonoman
    Mar 1 '15 at 4:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.