No error message in journactl of custom service

Question

I try to write a custom service file.

Unfortunately I see no output in the journalctl.

Here is my unit file:

root@om:~# cat /etc/systemd/system/ssh-tunnel-foo-de.service 
[Unit]
Description=Tunnel For ssh-tunnel-foo-de
After=network.target

[Service]
User=autossh
ExecStart=/usr/bin/ssh -o "ExitOnForwardFailure yes" -o "ServerAliveInterval 60" -N -R 1080:localhost:1080 [email protected]
ExecStartPre=-/usr/bin/ssh [email protected] "for pid in  $$(ps -u tunnel | grep sshd| cut -d' ' -f1); do kill -9 $$pid; echo kill old ssh process $$pid; done"
Restart=always
RestartSec=5s
StartLimitInterval=0

[Install]
WantedBy=multi-user.target 

Logoutput:

root@om:~# journalctl -u ssh-tunnel-foo-de
Apr 01 10:12:28 om systemd[1]: Stopped Tunnel For ssh-tunnel-foo-de.
Apr 01 10:12:28 om systemd[1]: Starting Tunnel For ssh-tunnel-foo-de...
Apr 01 10:12:28 om systemd[1]: Started Tunnel For ssh-tunnel-foo-de.
Apr 01 10:12:28 om systemd[1]: ssh-tunnel-foo-de.service: Main process exited, code=exited, status=217/USER
Apr 01 10:12:28 om systemd[1]: ssh-tunnel-foo-de.service: Unit entered failed state.
Apr 01 10:12:28 om systemd[1]: ssh-tunnel-foo-de.service: Failed with result 'exit-code'.

How can I debug what's wrong. I guess some stderr output gets lost.

Answer 1

Introduction

Your original question pertains to how you can get more output from systemd. Have a look at How to debug systemd unit ExecStart

But let's see if we can not get your service working first.

About systemd

Some issues

1. ssh is forking after it has started.
2. An ssh tunnel should run in the background
3. A systemd process is by default run as type "simple" - which expects that the command in StartExec is the main service (see "1")
4. For systemd to know if your service is alive or not a PIDFile is required a lot of the time. GuessMainPID defaults to yes, but might be, and is with regards to ssh, mistaken.
5. You kill the tunnel locally - no need to go abroad with a sword

This introduce a lot of complexity which is better handled by a wrapper script.

/etc/systemd/system/ssh-tunnel-foo-de.service

[Unit]
Description=Tunnel For ssh-tunnel-foo-de
After=network-online.target

[Unit]
Description=Tunnel For ssh-tunnel-foo-de
After=network-online.target

[Service]
User=autossh
ExecStart=/home/autossh/bin/ssh-tunnel.sh start
ExecStop=/home/autossh/bin/ssh-tunnel.sh stop
PIDFile=/home/autossh/bin/ssh-tunnel.pid
Restart=always
RestartSec=5s
StartLimitInterval=0
SuccessExitStatus=255
Type=forking

[Install]
WantedBy=multi-user.target

Explanations to systemd

The wrapper script saves the pid to PIDFile. If you change the location you must keep your service file and your wrapper script in sync.

You must set the "Type" to forking so that systemd knows that a fork is happening.

SuccessExitStatus - seems the process dies with 255 - so we handle that so that a stopped service is not listed as "failed" after it is stopped.

Wait until after the network-online.target is started. This means you actually have a network connection, not just a network management stack. https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ https://www.freedesktop.org/software/systemd/man/systemd.unit.html#

A wrapper script:

/home/autossh/bin/ssh-tunnel.sh

Of course you need to decide where you actually want to put this and fix paths accordingly.

#!/bin/bash
[email protected]

usage () {
    echo "usage: $0 {start|stop}"
    exit
}

cd $HOME/bin

case $1 in
    "start")
        /usr/bin/ssh -M -S socket-${MYHOST} -fnNT -o BatchMode=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=60 -R 1080:localhost:1080 ${MYHOST}
        EC=$? ;  [ $EC -ne 0 ] && exit $EC
        PID=$(/usr/bin/ssh -S socket-${MYHOST} -O check socket-${MYHOST} 2>&1 | awk '/Master running \(pid=/ { sub(/^.+pid=/,"") ; sub(")","") ; print }')
        echo $PID > $HOME/bin/ssh-tunnel.pid
        ;;
    "stop")
        /usr/bin/ssh -S socket-${MYHOST} -O exit ${MYHOST}
        /bin/rm -f $HOME/bin/ssh-tunnel.pid
        exit 0
        ;;
    *) usage ;;
esac

Some explanations:

Look up some of the parameters for ssh.

• -M and -S sets up a Master connection and a named Socket to control the master.
• f - go to the background
• n - Prevent reading from stdin (implied by -f)
• N - No remote command
• T - Disable pseudo-tty allocation

The "start" part sets up the tunnel and the "master". Then it queries the master for the pid of the tunnel and saves this to a pid-file to the benefit of systemd. If run as root you could save to /var/run/*pid.

The "stop" part connects to the master and issues an "exit". Then removes the pid file.

Credits to https://stackoverflow.com/a/15198031/2045924

Answer 2

The line User=autossh was the mistake.

This user did not exist.

After creating the user it worked.

The default Type=simple works nice.

There is no need for a wrapper script.