Unbound can't seem to read certificate files for DNS-over-TLS, gets "Permission denied"

Question

I'm trying to set up DNS-over-TLS (DoT) with unbound resolver. i.e. I'm trying to encrypt the connection between the client and unbound I'm NOT trying to encrypt the unbound resolver → upstream connection, which many guides on the internet are talking about.

I have the following in the config file, as explained in the man page, and also described here:

 server:
   interface: 0.0.0.0@853

   tls-port: 853
   tls-service-key: "/etc/letsencryp/live/DOMAIN/privkey.pem"
   tls-service-pem: "/etc/letsencryp/live/DOMAIN/fullchain.pem"

But when I try to restart unbound, I get the following permission denied on the certificate files.

package-helper[778]: /var/lib/unbound/root.key has content
package-helper[778]: success: the anchor is ok
unbound[813]: [1586107523] unbound[813:0] error: error for cert file: /etc/letsencryp/live/DOMAIN/fullchain.pem
unbound[813]: [1586107523] unbound[813:0] error: error in SSL_CTX use_certificate_chain_file crypto error:0200100D:system library:fopen:Permission denied
unbound[813]: [1586107523] unbound[813:0] error: and additionally crypto error:20074002:BIO routines:file_ctrl:system lib
unbound[813]: [1586107523] unbound[813:0] error: and additionally crypto error:140DC002:SSL routines:use_certificate_chain_file:system lib
unbound[813]: [1586107523] unbound[813:0] fatal error: could not set up listen SSL_CTX
systemd[1]: unbound.service: Main process exited, code=exited, status=1/FAILURE

I have tried moving the files out of this directory, and experimented with setting root or unbound as the owner. The only way I could make it work was to place the files directly in the /etc/unbound/ directory. A symlink in the same location pointing to letsencrypt managed files didn't work either. This is not ideal, as I'd need to regularly copy the certificate files out of letsencrypt directory whenever a certificate renewal occurs and/or unnecessarily restart the DNS resolver.

I have thoroughly checked that a chroot is not configured in config files, or default settings, or compiled in the binary. In fact, it has been explicitly disabled by default in Debian (bug report)

How can unbound be unable to read files, that is right there, with unbound:unbound as owner:group, and permissions set as readable?

I'm using unbound version 1.9.0-2+deb10u1 on Debian buster (10), if it's of any importance.

Answer 1

It's not a permission issue per se, as Unbound reads the files as root before dropping privileges. You can leave letsencrypt certificate files as root:root and they would still work.

What's preventing unbound from reading the certificates is AppArmor. Debian(and many other distros) restrict server binaries for security purposes. You can see the default AppArmor config at /etc/apparmor.d/usr.sbin.unbound, it doesn't allow reading outside of /etc/unbound, /var/lib/unbound/, /etc/openssl(through #include <abstractions/openssl> directive), and some other directories. Because your letsencrypt certificates are in another directory, it's not accessible to unbound, even when running as root.

You can work around this by allowing unbound to read the certificate files (and "live" links pointing to them). Here's a local override config for AppArmor, this file is included from the main file.

$ cat /etc/apparmor.d/local/usr.sbin.unbound
/etc/letsencrypt/archive/** r,
/etc/letsencrypt/live/** r,

Once you change this file, you'll need to activate the changes by running:

$ apparmor_parser --replace /etc/apparmor.d/usr.sbin.unbound

Once you follow these steps, unbound shouldn't have any problem in reading the key material.

You can get more information on AppArmor from Ubuntu's nice guide