2

TIL that docker kill means kill in the sense of "make it dead", as opposed to the POSIX sense of "send a signal".

We have several containers that we need to send a SIGHUP to in order to reload configuration, but this causes them to ignore their restart policy of "always", which isn't what we want.

What's the best way to send signals to these containers without affecting their ability to auto-restart?


To more clearly demonstrate the issue we're seeing, take the following example.

We have some container that has a restart policy set to always

$ docker inspect cloudwatch-exporter | jq .[].HostConfig.RestartPolicy
{
  "Name": "always",
  "MaximumRetryCount": 0
}

We reload the config at some point using docker kill:

$ docker kill --signal=SIGHUP cloudwatch-exporter
cloudwatch-exporter

Some time later, something happens that kills the process. To simulate this, I'll send a signal within the container:

$ docker exec cloudwatch-exporter bash -c "kill 1"

At this point, the container is dead and won't restart:

$ docker ps -a | grep cloudwatch-exporter
c7827204bba5        prom/cloudwatch-exporter:cloudwatch_exporter-0.8.0                                "java -jar /cloudwat…"   20 hours ago        Exited (143) 3 minutes ago                            cloudwatch-exporter

What alternatives do we have for using docker kill? For many cases docker exec works, but it won't work for any container that only contains a single static-linked binary.

1

With the --signal option to docker kill?

docker kill --signal=SIGHUP <container>

Or with docker exec?

docker exec <container> killall -HUP nginx
  • We're seeing docker kill interfere with the restart policy, which is what I'm trying to avoid. The kill and killall are usually shell builtins, so you'd have to wrap it in bash -c "killall -HUP nginx" or something. But that won't work with images that are single statically linked binaries. – leedm777 Jul 24 '20 at 18:57
  • Did you actually specify the signal? – Michael Hampton Jul 24 '20 at 19:01
  • Yes, we specified the signal. I edited the question to be more specific about what we're seeing. – leedm777 Jul 24 '20 at 19:06
  • You sent the signal correctly. Now you should investigate why the process is stopping. This focus on sending the signal seems like a red herring. – Michael Hampton Jul 24 '20 at 19:07
  • I'm not as concerned with why the process stopped as I am with why it didn't restart. FWIW, in the specific case I ran into today it was a startup timing issue with our deploy automation. We sent a SIGHUP to a container right after it started, and the container promptly died. My guess is that we sent the signal before it could register its signal handler. But that's irrelevant. Whether its a signal handler, segfault, or OOM, I'm expecting Docker to restart my container. The docker kill command stops that from working, so I'm looking for alternatives. – leedm777 Jul 24 '20 at 21:45
2

This is a known Docker bug.

See https://github.com/moby/moby/issues/11065 "Non-fatal signals break restart policies"

See https://github.com/moby/moby/issues/41302 "Signal breaks unless-stopped restart policy"

Bypass the problem by doing docker exec "kill -HUP 1".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.