3

this morning I woke up to a 'service down' email I have received from watchdog. After connecting to the server I have found the docker service down. After little investigation, I have found that this morning Ubuntu made an unattended upgrade of the containerd service and after the upgrade, docker service was not started automatically.

How can I make sure that after the upgrade this will not happen again?

1
2

I set Docker and its related packages to hold so they are only upgraded on planned upgrades.

sudo apt-mark hold docker containerd

When I want to upgrade Docker I run apt-mark unhold, upgrade the packages, and hold them again. I have an Ansible playbook to automate this process. My playbook to install and configure Docker sets the packages to hold directly after the installation.

3
  • Thanks for this. This acutally answers the question. The containerd upgrade fiasko was just an example of something that can happen. – Krystian Dec 1 '20 at 12:55
  • is your ansible playbook available somewhere, like ansible-galaxy, or whatever? – MaxiReglisse Dec 7 '20 at 15:49
  • No, it isn't, sorry. But it's pretty easy, you only need the dpkg_selections module for hold. – Gerald Schneider Dec 8 '20 at 6:59
1

We had the same issue. I opted for using a systemd unit file to create a loose dependency between containerd and dockerd.

Here's how:

As root:

Make the override directory (permission 755)

mkdir /etc/systemd/system/containerd.service.d/

Create the override file in the above directory (permissions 644)

vi override.conf

Add the following to the file

[Unit]
Before=docker.service
Wants=docker.service

Restart systemd daemon

systemctl daemon-reload

Check it works

Stop containerd

systemctl stop containerd

Check dockerd has stopped

systemctl status docker

Start containerd

systemctl start containerd

Check dockerd has started

systemctl status docker

++++++++++++++++++++++++++

Some links to other documentation on the subject

https://www.shellhacks.com/systemd-service-file-example/

https://man7.org/linux/man-pages/man5/systemd.service.5.html

0

The documentation on docker restart policies is helpful, start your containers with the always or unless-stopped options as appropriate for your use.

It also seems reasonable to stop Ubuntu from carrying out unattended upgrades.

3
  • That doesn't help in this case. I had the same problem this morning, after the upgrade of containerd the docker systemd unit was in the state disabled. – Gerald Schneider Dec 1 '20 at 7:38
  • That's a bug in the upgrade - it should be reported. – user602730 Dec 1 '20 at 7:39
  • It already is – Gerald Schneider Dec 1 '20 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.