37

I have a custom daemon that is managed by upstart on my Ubuntu server. It works perfectly except that I need to capture (log) the daemon's output. The official stanzas page says that I can use console logged to do this, but what file does it log to?

I've also read that console logged is no longer a valid stanza. I'm currently using 0.3.9 (Hardy) but will upgrade to 0.6.x (Lucid) in a few months. If console logged in fact won't work with later versions, what do I use instead?

1
  • 1
    Can you simply update your custom daemon to send the output to syslog, or to a logfile specified in the daemon's configuration file?
    – Zoredache
    Feb 18, 2010 at 2:22

7 Answers 7

36

This snippet will pipe the output of your service into logger, while still allowing you to exec the service process (thus replacing the shell process) so that upstart doesn't get confused. It also makes sure the logger process is reparented to init, so it's not a child of your service, and it avoids leaving cruft sitting around in the filesystem, even though it needs to create a fifo temporarily.

script
  mkfifo /tmp/myservice-log-fifo
  ( logger -t myservice </tmp/myservice-log-fifo & )
  exec >/tmp/myservice-log-fifo
  rm /tmp/myservice-log-fifo
  exec myservice 2>/dev/null
end script

Here's how it works:

  1. mkfifo /tmp/myservice-log-fifo simply makes the fifo special file (aka named pipe). Type man 7 fifo for more info.
  2. ( logger ... </tmp/myservice-log-fifo & ) starts logger reading from the fifo, in the background. The parens cause the logger process to be reparented to init, rather than remaining a child of the current shell process.
  3. exec >/tmp/myservice-log-fifo redirects the current shell's stdout to the fifo. Now we have an open file descriptor for that fifo, and we don't actually need the filesystem entry any more...
  4. rm /tmp/myservice-log-fifo so we'll remove it.
  5. exec myservice 2>/dev/null simply runs the service in the usual way. Stdout is already going to the fifo, and that won't change when the new program executes.

UPDATE: set -e is not needed as Upstart runs scripts with this option by default (see http://upstart.ubuntu.com/cookbook/#develop-scripts-using-bin-sh)

4
  • great answer, and doesn't leave fifo files lying around anywhere. Mar 29, 2012 at 12:25
  • What is the set -e? Sep 24, 2012 at 22:23
  • 1
    The set -e causes the script to exit immediately if any command fails. Without that line, the script would continue running subsequent commands uselessly (and possibly dangerously). Oct 13, 2012 at 23:53
  • 1
    To log stderr as well as stdout, add the line exec 2>&1 above the rm line, and remove the 2>/dev/null from the last line.
    – itsadok
    May 19, 2015 at 11:20
32

For recent Ubuntu versions (12.04+), simply use

console log

And the daemon output (STDOUT & STDERR) will be appended to /var/log/upstart/<service>.log

http://upstart.ubuntu.com/cookbook/#console-log

1
  • 2
    That would be clean and nice, but unfortunately it doesn't work on CentOS 6.x which still ships an ancient version of upstart that doesn't support this. Jun 15, 2017 at 10:31
11

If you use the console output stanza, and then pipe your script's output to logger (the shell command interface to the syslog(3) system log module) then that will work.

For example

console output
exec /my/script | logger

will log to /var/log/messages

For example

console output
exec /my/script | logger -t my-script

will log to /var/log/messages and tag each message with my-script

logger --help for logger usage options.

(I'm on the Amazon Linux AMI, which is Centos 5.x based; YMMV)

1
  • 4
    It turns out this isn't a good solution. upstart latches onto the PID of logger, not of the process you actually want it to manage. Jun 16, 2015 at 22:39
11

I did not get the mkfifo trick to work satisfactorily; it did not seem to capture stderr, and attempts to redirect caused Upstart to bail with no errors.

It also has an unfortunate side effect of making the logger process hang around as a child of init, so the information about who "owns" the logger is lost, and anyone not already aware of the mkfifo might assume it's a dangling process that can be killed.

Instead I ended up with the following solution, which resolves all of these issues. It causes logger to become a child process, while preserving the service as the root process. Unfortunately, it requires exec'ing bash, but it just looks dirty.

script
  # ... setup commands here, e.g. environment, cd, ...
  exec bash <<EOT
    exec 1> >(logger -t myservice) 2>&1
    exec myservice
EOT
end script

This uses a trick that redirects stdout and stderr to a command. Since we exec the service inside the bash command, this has the side effect of replacing the shell and magically making bash become a child process of the service, as shown by ps aufxw:

myservice 
 \_ bash -c exec 1> >(logger -t myservice) 2>&1 && exec myservice
    \_ logger -t myservice

For some reason the above command has to be wrapped in a bash -c. I assume this is because Upstart only pretends to run your script via Bash, but isn't actually. If anyone can suggest a way to avoid the extra bash shell, that would be awesome.

1
  • 1
    Yes! I'm so glad I found this. Looks better than the mkfifo variant and in my case I needed to exec bash -l << EOF so no loss there.
    – thom_nic
    Mar 10, 2015 at 14:19
6

This is ugly but so far the best I have found

exec /path/to/server >> /tmp/upstart.log 2>&1

1
  • 2
    The solution is not a good when it comes to log rotation, because the application is logging directly to the file. Logging through syslog avoids related problems. Jun 29, 2015 at 2:05
3

You can also redirect the output to syslog, e.g.

exec $SERVER 2>&1 | logger -t myservice -p local0.info

However, the pipeline may cause upstart to confuse the PID of the logging process with the PID of the daemon.

2
  • The daemon confusing the PID is actually a useless daemon, since it is supposed to watch the process, respawn it then. Any idea on how to make sure the right PID is being watched? Feb 10, 2012 at 11:04
  • I think (but haven't tried) that you'll want the expect fork or expect daemon stanza. Or you could cat the pid-file into the log message otherwise, I guess. Jun 20, 2012 at 12:45
1

Another alternative is using tee like:

exec $SERVER 2>&1 | tee /dev/stderr | logger -t myservice

to get both upstart file, and syslog output

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .