2

We are running Docker swarm mode in production, having 4 nodes, 3 of them share the manager status. We've noticed that once in a few days all the containers on one of the hosts are got restarted.

I've looked into syslog at that time and could see the following messages (repeating for all the containers that are scheduled on this host):

Jan 26 07:26:14 HOST0 dockerd[13104]: time="2019-01-26T07:26:14.954777646Z" level=warning msg="failed to deactivate service binding for container service_container.1.lhc0gejxgb8y340bg9o2wfcm2" error="No such container: service_container.1.lhc0gejxgb8y340bg9o2wfcm2" module=node/agent node.id=2g08blfds9z26ja2ou06pv2zl

There are some swarm membership-related messages preceding these but they also happen whithout issues and anyway are at level=info.

It's important to mention that on certain hosts we have single-instance stateful services like databases.

So far I need to understand what triggers the recreation of the services and how to avoid it. Is there anything special I shall grep the logs for, to start with?

Thanks a lot in advance!

1 Answer 1

0

I suggest starting with the tasks of the service that restarted. It should give a reason for exiting, and the exit code of the container's PID 1 process.

docker service ps $SERVICE_NAME will list the last 5 service tasks (or whatever your history limit is set to).

Take the ID of the exited task and use docker inspect $TASK_ID to get the details.

2
  • 1
    Seems that all the tasks on the certain node get terminated. So they did not die by themselves, it was docker that terminated all of them.
    – saabeilin
    Feb 7, 2019 at 7:54
  • I have the same issue in 2022. All of my swarm services (~30) got restarted at the same time and their logs just say "Shutting down" because of regular sigterm/sigint. I thought the server restarted, but no. Docker itself didn't restart either. Apr 8, 2022 at 18:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .