Publish network services using Systemd machinectl

Question

TL;DR: How to expose network services to the host through machinectl/nspawn containers?

I'm trying to make things perfectfor my setup, i.e. installing my services in separated chroot instances.

To start them, I want to comply with SystemD management with systemd-nspawn and machinectl:

# debootstrap stretch /var/lib/machines/mymachine
# machinectl start mymachine # works well!
# machinectl shell root@mymachine bash

Then I install dummy apache2 on it:

 (jail)# apt install -y apache2 && systemctl enable apache2 && systemctl start apache2
 (jail)# ss -tnlp | grep 80 # yields apache2 running.

However, the 80 port (or even any port whatsoever, say 8080) is not visible on listening on the host. How to make it work ?

Thank you team,

Answer 1

You need an nspawn file for the machine that exposes the port.

[Network] Section Options

Port= Exposes a TCP or UDP port of the container on the host. This option corresponds to the --port= command line switch, see systemd-nspawn(1) for the precise syntax of the argument this option takes. This option is privileged (see above).

From systemd-nspawn(1):

-p, --port= If private networking is enabled, maps an IP port on the host onto an IP port on the container. Takes a protocol specifier (either "tcp" or "udp"), separated by a colon from a host port number in the range 1 to 65535, separated by a colon from a container port number in the range from 1 to 65535. The protocol specifier and its separating colon may be omitted, in which case "tcp" is assumed. The container port number and its colon may be omitted, in which case the same port as the host port is implied. This option is only supported if private networking is used, such as with --network-veth, --network-zone= --network-bridge=.

The [email protected] unit uses --network-veth

So something like

[Network]
Port=80

should work.

Answer 2

I just run systemd-nspawn without any network flags, it uses host networking - then route my web app's port using nginx.

Assume $MACHINE_NAME is webapp in this instance:

systemd-nspawn -D /var/lib/machines/webapp -M webapp --chdir=/ ./entrypoint &

• -D is the directory of the filesystem root,
• -M is the machine name,
• --chdir changes directory inside the container and
• ./entrypoint is a script I want to run (this can be any command, script, etc.)
• & runs it in the background (so you can do other things in the particular tty you ran the command in).

The entrypoint script was originally from a podman container this filesystem was exported from. Here's what it's doing (it's very simple):

#!/usr/bin/env bash
cd /home/localuser/webapp
npm run start

So this just runs the web app I developed inside the container. It's designated to run on port 3000.

Then, nginx is set up to proxy 80 for 3000, which has an open port in the firewall:

# cat /etc/nginx/sites-available/webapp
server {
    listen 80;
    server_name webapp.xyz;

    location / {
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header   Host $http_host;
        proxy_pass         "http://127.0.0.1:3000";
    }
}

You can see it's using the loopback device - but any of your host networking is available inside the container when systemd-nspawn is run without --boot, and without --network-{bridge,interface,ipvlan,macvlan,veth}. It's very similar to chroot this way.

Then, once systemd-nspawn is running (in the background because & is at the end of the command), one can run:

machinectl enable $MACHINE_NAME

and it will generate a symlink with the name of the machine from an .nspawn template.

.nspawn template is here: /lib/systemd/system/[email protected]

$MACHINE_NAME.nspawn link will be added here: /etc/systemd/system/machines.target.wants/systemd-nspawn@$MACHINE_NAME.service

If you want to default to running new machines using host networking, just backup the original and remove some of the runtime arguments:

/* backup original unit file */

# mv /lib/systemd/system/[email protected] \
     /lib/systemd/system/[email protected]

/* echo a more simple file into its place */

# echo '[Unit]
Description=Container %i
Documentation=man:systemd-nspawn(1)
[email protected]
PartOf=machines.target
Before=machines.target
After=systemd-resolved.service [email protected]
RequiresMountsFor=/var/lib/machines/%i

[Service]
ExecStart=systemd-nspawn --quiet --keep-unit --link-journal=try-host --machine=%i --chdir=/ ./entrypoint
KillMode=mixed
Type=notify
RestartForceExitStatus=133
SuccessExitStatus=133
Slice=machine.slice
Delegate=yes
TasksMax=16384
WatchdogSec=3min

DeviceAllow=char-pts rw
DeviceAllow=/dev/loop-control rw
DeviceAllow=block-loop rw
DeviceAllow=block-blkext rw

# these two only nec. for LUKS - can remove 
DeviceAllow=/dev/mapper/control rw
DeviceAllow=block-device-mapper rw

[Install]
WantedBy=machines.target' > /lib/systemd/system/[email protected]

You can see here the command to run inside the unit file is essentially the same, I just added the flag --link-journal=try-host, which places a log at /var/log/journal/$MACHINE_UUID for monitoring the container.

Since this is a template, it assumes I'll be invoking a script named entrypoint from the root dir of each container, which gives them more of a 'command invoking container' behavior (e.g. Docker-style as opposed to LXC-style).

Notes about issues:

I'd like to use --user flag, but it appears to be non-functional ATM. There's some talk about a regression here regarding the flag, but it pertains to v237 on CentOS, so not sure if it relates to 241 on Ubuntu 20.04, which is what I'm using ATM. Anyhow, containers will run w/ root UID for now...

systemd-networkd is supposed to work with --network-veth to create a DHCP host on host0 and client on ve-$MACHINE_NAME (limit 16 chars) - IF both the host and the client use systemd-networkd. However, I have not been able to get this to work properly in practice on Ubuntu 20.04. I'm guessing there's a bug re: this functionality in my version of systemd and will be doing further testing on later versions.

Environment info:

# cat /etc/os-release && systemd --version
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
systemd 245 (245.4-4ubuntu3.6)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

Further reading:

• man systemd-nspawn
• man systemd.nspawn (.nspawn file spec)
• man machinectl
• arch wiki on systemd-nspawn