Does Docker Swarm migrate resource hungry containers to other nodes?

Question

Let's assume a 3 node Docker cluster. One one of these nodes, resources are tight. A couple of services which can be quite greedy in terms of CPU or RAM happened to start on the same host.

Will Docker at any point check if the other nodes have less work to do, and migrate one or more services from the pressured node to one of the less pressured nodes?

Is there any built-in functionality in Docker to handle this, or will a migration eventually be a result of e.g. an OOM killer taking out a container?

Answer 1

Swarm does not move tasks (containers). Its goal there is "do no harm" by moving containers that you didn't explicitly tell it to do so. If you do a service update it will re-create the container, potentially on other nodes based on 1. ensuring tasks in that service are spread out across nodes and 2. container density per node.

But AFAIK that assessment isn't based on moment-by-moment resource utilization.

You can use --limit-cpu/memory and --reserve-cpu/memory options on your service create/update commands that will help. reserve will cause Swarm to track that reservation and if you set reservations for multiple services it will make sure they are scheduled on nodes that have those available resources, but again, it just uses the reservation table, not real-time utilization to determine that (AFAIK).

I've seen people use Prometheus to monitor and kick off alerts that scale a service, so it's feasible those alerts could also kick off moving a container.