Upstart Job as Unprivileged user fails with permission denied

Question

I'm trying to set up a Sage Cell Server (an open source mathematical engine) running as an unprivileged user. I've created the unprivileged user sagecell to do the job. It is installed correctly and, if I run it normally using

sagecell@primary-server:~/sage/devel/sagecell$ ../../sage web_server.py

then everything works as expected. An interesting requirement of this application (that may or may not be relevant) is that it requires a passwordless ssh login (i.e., ssh localhost works). I've set up the unprivileged account with an ssh-key to do the job.

Now, rather than manually making a screen session after booting up the server and running it in a screen session, I tried making an upstart job with this setup:

description "Sage Cell Server Starter"

start on runlevel [2345]
stop on starting rc runlevel=[016]

kill timeout 5

exec su -s /bin/sh -c 'exec "$0" "$@"' sagecell -- /home/sagecell/sage/sage /home/sagecell/sage/devel/sagecell/web_server.py

This, however, does not work. Looking at /var/log/upstart/sagecell.log gives the following error:

2013-02-20 00:11:09,665 sagecell:INFO starting tornado web server
Traceback (most recent call last):
  File "/home/sagecell/sage/devel/sagecell/web_server.py", line 78, in <module>
    application = SageCellServer()
  File "/home/sagecell/sage/devel/sagecell/web_server.py", line 57, in __init__
    kernel_timeout=kernel_timeout)
  File "/home/sagecell/sage-5.7.rc0/devel/sagecell-main/trusted_kernel_manager.py", line 28, in __init__
    self._sender = sender.AsyncSender() # Manages asynchronous communication
  File "/home/sagecell/sage-5.7.rc0/devel/sagecell-main/sender.py", line 22, in __init__
    self.router.bind(self.filename)
  File "socket.pyx", line 489, in zmq.core.socket.Socket.bind (zmq/core/socket.c:4797)
zmq.core.error.ZMQError: Permission denied

EDIT: It should be noted that the application binds to port 8080 (and only 8080 as far as I'm aware). The important thing is that it works without privileges when I run it but not when Upstart runs it as the same user.

Can anyone tell me why an application that works perfectly fine under the user sagecell fails to run under an upstart job running the script AS sagecell?

Answer 1

The start condition with only runlevel specified:

start on runlevel [2345]

happens very early in the boot process. At that point there usually is no network device available. Also, many other services have not yet started.

As you can see from this log line:

File "socket.pyx", line 489, in zmq.core.socket.Socket.bind (zmq/core/socket.c:4797) zmq.core.error.ZMQError: Permission denied

it can not bind to a socket, which indicates that the network device is not yet up.

What you would do is adjust the start condition so that the service starts later when there are enough infrastructure available.
This is what I have used in a situation where I needed to start a web service with upstart:

start on (runlevel [2345] and local-filesystems and net-device-up IFACE!=lo)

This ensures that your service starts only when local filesystems are available and when a network connection comes up. Here it is defined also that a local loopback device is not enough for a network connection. The stanza requires that a real network interface is up, like eth0.

EDIT

Then there may be default directory issue. Upstart runs the started programs with default directory set to /. Application started with su probably does not have enough privileges to write its files (log files, etc.)

Try adding following stanza:

chdir /home/sagecell/sage/devel/sagecell

This changes the default directory to a place where sagecell user has write permissions.

Answer 2

The "starting tornado web server" indicates it starts its own web server, on what port? If the normal port 80, only root can do that. Look at http://www.itp.uzh.ch/~dpotter/howto/daemonize, check if Pyhon has some guidelines for this. What does the Sage documentation recommend? It's not a fringe package by any means, I'd assume they suggest something for a mainstream distribution like Ubuntu (if it isn't already packaged up for use, that is).

Answer 3

If you are running upstart 1.4 or newer, you can use the setuid stanza rather than using su:

setuid sagecell

Regarding the environment upstart runs your job in, see:

• http://upstart.ubuntu.com/cookbook/#checking-how-a-service-might-react-when-run-as-a-job
• http://upstart.ubuntu.com/cookbook/#see-the-environment-a-job-runs-in

Note specifically that even variables such as $HOME will not be set by default when running a system job as any user. As an alternative you might want to consider a User Job:

• http://upstart.ubuntu.com/cookbook/#user-job

If you do wish to use GNU Screen, see:

• http://upstart.ubuntu.com/cookbook/#run-an-application-through-gnu-screen