Upstart script vs. command-line: why this difference in behavior?

Question

I'm trying to set up Tomcat to start with upstart. I find the following works:

  description "Tomcat Server"

  start on runlevel [2345]
  stop on runlevel [!2345]
  respawn
  respawn limit 10 5

  setuid tomcat

  env JAVA_HOME=/usr/lib/jvm/default-java
  env CATALINA_HOME=/opt/apache-tomcat-7.0.34

  script
    chdir $CATALINA_HOME
    exec $CATALINA_HOME/bin/catalina.sh run
  end script

But if I remove the chdir, tomcat starts, but with plenty of FileNotFound exceptions when my .wars load. (That is: the .wars themselves do load but they throw exceptions on load.)

Note this behavior is different from what I see when I invoke catalina.sh from the command line. Invoking from the command line, I can run /opt/apache-tomcat-7.0.34/bin/catalina.sh run from any directory (without chdiring) and everything's fine.

So why is the chdir necessary in my upstart script? (How is the upstart environment different from my command-line environment?)

---

Example of the errors I'm seeing w/ upstart when I don't chdir:

Feb 22, 2013 3:00:11 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive my-war.war
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: my-war.log (Permission denied)
    at java.io.FileOutputStream.open(Native Method)

Answer 1

This is a write permission issue. User tomcat is not allowed to create a log file to a location where it tries to create them. The location it tries to create log files in differs depending on how you start tomcat.

This piece from from your log caught my attention:

java.io.FileNotFoundException: my-war.log (Permission denied) at java.io.FileOutputStream.open(Native Method)

It says that the user (tomcat) is not allowed to create the file my-war.log. Here are the different scenarios:

Upstart with chdir

Upstart first chdirs to $CATALINA_HOME. User tomcat is allowed to create files there. So everything works.

Upstart without chdir

Upstart runs as root, so the default directory is /. User tomcat is not allowed to create files there. So you get permission denied errors.

Running tomcat from your home dir

Now you run tomcat as yourself from your home dir. You have write permission to you own directory. So everything works again.

Answer 2

I don't have access to an upstart version with setuid support, but this is what I would do in your case.

It looks like an environment discrepancy issue. Most likely an environment variable that is not set when running from upstart. Could it be that setuid does not set $USER or $HOME?

Therefore, I would recommend that you compare the environments. A easy, simple way would be to modify your init script like this and restart the job.

  script
    env > /tmp/env-upstart.log
    chdir $CATALINA_HOME
    exec $CATALINA_HOME/bin/catalina.sh run
  end script

Then also run env > /tmp/env-console.log when you normally would start it from the console (if you are using sudo, then do it with sudo env).

Then compare the two /tmp/env-upstart.log and /tmp/env-console.log files (sort them and open them with vimdiff) and it should be easy to find which variable in env-upstart is missing from the other file (or is set to something that you would not expect).

Update 1

And if you are still getting errors, I would check the following:

• User permissions: effective/real user ids might be differing. Compare the output of id, id -r -u, id -r -g, id -r -G, id -u, id -g, and id -G in both cases.
• Process limits: It could be that your upstart scripts have stricter limits. Try putting an ulimit -a to compare.
• Shell oddness: Not likely, but upstart uses sh while your user's shell is most likely bash or zsh. Try running the successful command from sh.

• Debug the script: If you are still really, really stuck, run catalina.sh in debug mode. This is as easy as running it from upstart like this:

  exec /bin/sh -x $CATALINA_HOME/bin/catalina.sh run 2>/tmp/catalina-upstart.log
  

  Then you can compare the two debug logs and maybe spot where the two scripts did something different.

Answer 3

Yes, it's about permission. But not related to Upstart. This is because in 'catalina.sh start', i.e. in the 'start' part, the real command is:

  ...
  -Djava.io.tmpdir="\"$CATALINA_TMPDIR\"" \
  org.apache.catalina.startup.Bootstrap "$@" start \
  >> "$CATALINA_OUT" 2>&1 "&"

Here the output log is fully assigned. While there is no "$CATALINA_OUT" in the RUN part. This will lead the writing permission problem mentioned above. Namely, if no specific output file path assigned, the output will go to either logs file or do that as your.war coded. These kinds of unclear log assignment will lead uncertain permission issues.

Answer 4

To get a complete dump of all environmental elements between running your server on the command-line compared to running it under Upstart, try installing my procenv tool, running it in both environments and diffing the output files:

• https://launchpad.net/procenv/
• http://ifdeflinux.blogspot.co.uk/2012/10/procenv-and-process-environment.html

This is available in the ubuntu raring and debian sid archives (and FreeBSD).