Start N processes with one systemd service file

Question

I found this systemd service file to start autossh to keep up a ssh tunnel: https://gist.github.com/thomasfr/9707568

[Unit]
Description=Keeps a tunnel to 'remote.example.com' open
After=network.target

[Service]
User=autossh
# -p [PORT]
# -l [user]
# -M 0 --> no monitoring
# -N Just open the connection and do nothing (not interactive)
# LOCALPORT:IP_ON_EXAMPLE_COM:PORT_ON_EXAMPLE_COM
ExecStart=/usr/bin/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -p 22 -l autossh remote.example.com -L 7474:127.0.0.1:7474 -i /home/autossh/.ssh/id_rsa

[Install]
WantedBy=multi-user.target

Is there a way to configure systemd to start several tunnels in one service.

I don't want to create N system service files, since I want to avoid copy+paste.

All service files would be identical except "remote.example.com" would be replace with other host names.

1.5 year later ...

I asked this question roughly 1.5 year ago.

My mind has changed. Yes, it's nice, that you can do this with systemd, but I will use configuration-management in the future.

Why should systemd implement a template language and substitute %h? .. I think it makes no sense.

Several months later I think this looping and templating should be solved on a different level. I would use Ansible or TerraForm for this now.

Answer 1

Well, assuming that the only thing changing per unit file is the remote.example.com part, you can use an Instantiated Service.

From the systemd.unit man page:

Optionally, units may be instantiated from a template file at runtime. This allows creation of multiple units from a single configuration file. If systemd looks for a unit configuration file, it will first search for the literal unit name in the file system. If that yields no success and the unit name contains an "@" character, systemd will look for a unit template that shares the same name but with the instance string (i.e. the part between the "@" character and the suffix) removed. Example: if a service getty@tty3.service is requested and no file by that name is found, systemd will look for getty@.service and instantiate a service from that configuration file if it is found.

Basically, you create a single unit file, which contains a variable (usually %i) where the differences occur and then they get linked when you "enable" that service.

For example, I have a unit file called /etc/systemd/system/autossh@.service that looks like this:

[Unit]
Description=AutoSSH service for ServiceABC on %i
After=network.target

[Service]
Environment=AUTOSSH_GATETIME=30 AUTOSSH_LOGFILE=/var/log/autossh/%i.log AUTOSSH_PIDFILE=/var/run/autossh.%i.pid
PIDFile=/var/run/autossh.%i.pid
#Type=forking
ExecStart=/usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC %i

[Install]
WantedBy=multi-user.target

Which I've then enabled

[user@anotherhost ~]$ sudo systemctl enable autossh@somehost.example.com
ln -s '/etc/systemd/system/autossh@.service' '/etc/systemd/system/multi-user.target.wants/autossh@somehost.example.com.service'

And can intereact with

[user@anotherhost ~]$ sudo systemctl start autossh@somehost.example.com
[user@anotherhost ~]$ sudo systemctl status autossh@somehost.example.com
autossh@somehost.example.service - AutoSSH service for ServiceABC on somehost.example
   Loaded: loaded (/etc/systemd/system/autossh@.service; enabled)
   Active: active (running) since Tue 2015-10-20 13:19:01 EDT; 17s ago
 Main PID: 32524 (autossh)
   CGroup: /system.slice/system-autossh.slice/autossh@somehost.example.com.service
           ├─32524 /usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC somehost.example.com
           └─32525 /usr/bin/ssh -L 40000:127.0.0.1:40000 -R 40000:127.0.0.1:40001 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC somehost.example.com

Oct 20 13:19:01 anotherhost.example.com systemd[1]: Started AutoSSH service for ServiceABC on somehost.example.com.
[user@anotherhost ~]$ sudo systemctl status autossh@somehost.example.com
[user@anotherhost ~]$ sudo systemctl status autossh@somehost.example.com
autossh@somehost.example.com.service - AutoSSH service for ServiceABC on somehost.example.com
   Loaded: loaded (/etc/systemd/system/autossh@.service; enabled)
   Active: inactive (dead) since Tue 2015-10-20 13:24:10 EDT; 2s ago
  Process: 32524 ExecStart=/usr/bin/autossh -M 40000 -NR 5000:127.0.0.1:5000 -i /opt/ServiceABC/.ssh/id_rsa_ServiceABC -l ServiceABC %i (code=exited, status=0/SUCCESS)
 Main PID: 32524 (code=exited, status=0/SUCCESS)

Oct 20 13:19:01 anotherhost.example.com systemd[1]: Started AutoSSH service for ServiceABC on somehost.example.com.
Oct 20 13:24:10 anotherhost.example.com systemd[1]: Stopping AutoSSH service for ServiceABC on somehost.example.com...
Oct 20 13:24:10 anotherhost.example.com systemd[1]: Stopped AutoSSH service for ServiceABC on somehost.example.com.

As you can see, all instances of %i in the unit file get replaced with somehost.example.com.

There's a bunch more specifiers that you can use in a unit file though, but I find %i to work best in cases like this.

Answer 2

Here is a python example, which was what I was looking for. The @ in the service filename lets you start N processes:

$ cat /etc/systemd/system/my-worker@.service

[Unit]
Description=manages my worker service, instance %i
After=multi-user.target

[Service]
PermissionsStartOnly=true
Type=idle
User=root
ExecStart=/usr/local/virtualenvs/bin/python /path/to/my/script.py
Restart=always
TimeoutStartSec=10
RestartSec=10

Various methods to call it

Enabling various counts for example:

• Enable 30 workers:

  sudo systemctl enable my-worker\@{1..30}.service
  

• Enable 2 workers:

  sudo systemctl enable my-worker\@{1..2}.service
  

Then be sure to reload:

sudo systemctl daemon-reload

Now you can start/stop then in various ways:

• Start 1:

  sudo systemctl start my-worker@2.service
  

• Start Multiple:

  sudo systemctl start my-worker@{1..2}
  

• Stop Multiple:

  sudo systemctl stop my-worker@{1..2}
  

• Check status:

  sudo systemctl status my-worker@1
  

UPDATE: To manage instances as one service, you can do something like this:

/etc/systemd/system/some-worker@.service:

[Unit]
Description=manage worker instances as a service, instance %i
Requires=some-worker.service
Before=some-worker.service
BindsTo=some-worker.service

[Service]
PermissionsStartOnly=true
Type=idle
User=root
#EnvironmentFile=/etc/profile.d/optional_envvars.sh
ExecStart=/usr/local/virtualenvs/bin/python /path/to/my/script.py
TimeoutStartSec=10
RestartSec=10

[Install]
WantedBy=some-worker.service

/usr/bin/some-worker-start.sh:

#!/bin/bash
systemctl start some-worker@{1..10}

/etc/systemd/system/some-worker.service:

[Unit]
Description=manages some worker instances as a service, instance

[Service]
Type=oneshot
ExecStart=/usr/bin/sh /usr/bin/some-worker-start.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

And now you can manage all instances with sudo systemctl some-worker (start|restart|stop)

Here is some boilerplate for your script.py:

#!/usr/bin/env python

import logging


def worker_loop():
    shutdown = False
    while True:

        try:
            if shutdown:
                break

            # Your execution logic here.
            # Common logic - i.e. consume from a queue, perform some work, ack message
            print("hello world")

        except (IOError, KeyboardInterrupt):
            shutdown = True
            logging.info("shutdown received - processing will halt when jobs complete")
        except Exception as e:
            logging.exception("unhandled exception on shutdown. {}".format(e))


if __name__ == '__main__':
    worker_loop()

Answer 3

GregL's answer helped me a great deal. Here is an example of a unit template I used in my code using the example above for a gearman job server. I made a shell script that lets me create X amount of "workers" using this one template.

[Unit]
Description=az gearman worker
After=gearman-job-server.service

[Service]
PIDFile=/var/run/gearman_worker_az%i.pid
Type=simple
User=www-data
WorkingDirectory=/var/www/mysite.com/jobs/
ExecStart=/usr/bin/php -f gearman_worker_az.php > /dev/null 2>&1
Restart=on-success
KillMode=process

[Install]
WantedBy=multi-user.target

Answer 4

I have searched for solution to a similar task and actually have found one, that i believe is easier to accomplish, but is supposed to be hacky. (and is not mentioned here)

I needed to create multiple ssh connections after vpn creates a tunnel, so i have created a service that is dependent on tun device and calls shell script with appropriate commands.

Service /etc/systemd/system/ssh_tunnel.service:

[Unit]
Description=Reverse SSH Service to access hidden services
ConditionPathExists=|/usr/bin
Wants=sys-devices-virtual-net-tun0.device
After=network.target sys-devices-virtual-net-tun0.device

[Service]
Type=forking
ExecStart=/bin/sh /etc/openvpn/ssh_tunnels.sh 
RemainAfterExit=yes
TimeoutSec=0
GuessMainPID=no

[Install]
WantedBy=multi-user.target

/etc/openvpn/ssh_tunnels.sh:

!/bin/bash
#sleep 15

echo 'Tunelling some ports'
killall -HUP ssh

su - user -c 'ssh -f admin@172.171.1.1 -p 9999 -L :3690:svn.newbox.ru:3690 -L :8888:10.1.20.55:80 -L :8181:10.1.10.10:80 -N -vvv'

ssh -i /home/user/.ssh/id_rsa -f admin@172.171.1.1 -p 9999 -L :587:mail.domain.ru:587 -L :995:mail.newbox.ru:995 -L :22:10.1.2.1:22 -N -vvv &

exit 0

Result:

# systemctl status ssh_tunnel.service
● ssh_smartex.service - Reverse SSH Service to access hidden services
     Loaded: loaded (/etc/systemd/system/ssh_tunnel.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2020-03-20 16:01:07 UTC; 22min ago
    Process: 156 ExecStart=/bin/sh /etc/openvpn/ssh_tunnel.sh (code=exited, status=0/SUC>
      Tasks: 2 (limit: 4915)
     Memory: 3.8M
     CGroup: /system.slice/ssh_tunnel.service
             ├─166 ssh -f admin@172.171.1.1 -p 9999 -L :3690:svn.newbox.ru:3690 -L :8888:10.1.20.55:80 ->
             └─168 ssh -i /home/user/.ssh/id_rsa -f admin@172.171.1.1 -p 9999 -L :587:mail.newbox.ru:5>
...

However, i haven't checked yet, how it survives vpn restart, but it's another topic.