41

We have a task that loads some configuration files from an external data source. After the settings are uploaded we would like to be able to restart all the tasks in a service so that the settings propagate to all instances.

What's the best way to restart all services?

We have a 'workaround' that involves setting the 'number of tasks' to 0 and then back up, but this is definitely not how it's supposed to be done and has downtime.

1

11 Answers 11

51

Using the AWS CLI tool:

aws ecs update-service --force-new-deployment --service my-service --cluster cluster-name
2
  • 6
    As of 24th June 2020, the command should be aws ecs update-service --force-new-deployment --cluster CLUSTER_NAME --service SERVICE_NAME Jun 23, 2020 at 16:04
  • Thanks for the note! Amended the answer accordingly.
    – Ben Whaley
    Sep 29, 2020 at 20:37
12

What you're wanting to do is essentially the same as redeploying the Service.

To redeploy the service without downtime:

  1. Register a new Task Definition based on the current Task Definition (with the same details)
  2. Call UpdateService, associating the existing Service with the new Task Definition.

This should launch new Tasks for the new Task Definition and then kill the old Tasks for the old Task Definition, effectively restarting the tasks without downtime.

See: UpdateService

3
  • 1
    I needed to do this via the AWS Console, and this is the easiest way—you can manage the whole process manually if you need to. Helpful when you need to quickly relaunch all the tasks and don't have something more robust set up for the process — in the UI, go to the Task definition, create a new revision, update the service, then after a little time all the Tasks are relaunched! Jul 6, 2017 at 22:02
  • 2
    They've added a checkbox to the service update "Force new deployment" which lets you skip step 1 in your process. Mar 19, 2018 at 14:22
  • The comment about the "Forces new deployment" option was the Accepted Answer for myself.
    – ecbrodie
    Jun 15, 2018 at 19:55
6

this worked for me:

aws ecs list-tasks --cluster <cluster_name> | jq -r ".taskArns[]" | awk '{print "aws ecs stop-task --cluster <cluster_name> --task \""$0"\""}' | sh

the tasks then recreate on the same instances.

if you need new instances, then use this:

aws ecs list-services --cluster <cluster_name> | jq -r ".serviceArns[]" | awk '{print "aws ecs update-service --cluster <cluster_name> --force-new-deployment  --service \""$0"\""}' | sh
1
  • That second one seems to do something other than start new instances.
    – nroose
    Nov 10, 2018 at 7:25
2

Task as building block of ECS can be stopped by StopTask call. Service is composed of underlying tasks which may be stopped with same API call. Only missing part here is foreach around results from ListTasks call with defined family parameter. I wrote simple Lambda function which can help you with this.

2

I'm expanding on @user326608's answer above (thanks for the insight!).

This will restart ALL TASKS FOR ALL SERVICES FOR A CLUSTER by stopping all of its tasks. Each service will then automatically launch X number of new tasks, where X is the service's desired task count.

#!/bin/bash

index=0
taskArn=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --query "taskArns[${index}]" --output text)

until [ "$taskArn" = "None" ]
do 
  aws ecs stop-task --cluster ${CLUSTER_NAME} --task $taskArn
  ((index++))
  taskArn=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --query "taskArns[${index}]" --output text)
done
1
  • Note: If you want to restart tasks for a single service, simply force a new deployment as @Ben Whaley has described.
    – sudo soul
    Jul 7, 2019 at 0:39
1

If you are looking for how to restart all tasks of a service using the web console:

  • in the service details page, click on the blue "Update" button in the top right,
  • check the "Force new deployment" checkbox
  • click on the blue "Next Step" button at the bottom and go over the rest of the pages without making any changes
  • click on "Update service"

New tasks will be spawned and then the old ones will be stopped.

0

Based on the Amazon documentation it seems like you should be able to script the operations in question using the UpdateService API calls. There are some code samples available at the previous link, it seems possible that you should be able to adapt. It seems like writing a script to take care of reloading the services using the appropriate task definition after the updates to task configurations would be the most elegant solution to the problem.

There is more documentation about using AWS CLI with ECS which seems like it would be the easiest way to deal with batch scripting the restarting of services.

1
  • I can work on writing and posting the script/command sequence but I do not currently have access to an AWS account I would be able to use for testing this sort of thing, so it would be a rough draft/starting point since I would not be able to effectively test it...
    – Matt
    Jul 14, 2015 at 16:16
0

Works great https://github.com/fdfk/ecsServiceRestart

python ecsServiceRestart.py restart --services="app app2" --cluster=test
0

I have been working on this. It would pretty useful to be able to restart one task at a time reliably. The script below is what I am using now. It is pretty cautious. Requires you hit return for each task. There is a command to wait for the service to be stable, but that does not mean that the task is healthy. And I could in put a time delay. But in the end if things are going bad, the script would just slowly kill the app. So...

#!/bin/bash

if [ $# -eq 2 ]
then
    cluster=$1
    service=$2
else
    echo "Usage: $0 <cluster> <service>"
    exit 1
fi

echo
echo "Restarting $cluster $service tasks:"
echo

for task in $(aws ecs list-tasks --cluster $cluster --service-name $service | awk '{print $2}')
do
    echo
    echo -n "Press enter to stop $task"
    read -r
    echo
    echo "stopping $task..."
    aws ecs stop-task --cluster "$cluster" --task "$task"
    echo
    # aws ecs wait services-stable --cluster "$cluster" --services "$service"    done
0

I have a python boto3 script that does the ff:

  1. create a list of tasks with 'RUNNING' status for a service via

ecs_client.list_tasks(cluster=mycluster,serviceName=myservice,desiredStatus='RUNNING')

  1. do a for loop for the list of tasks above and stop each via

ecs_client.stop_task(cluster=mycluster,task=mytask)

  1. describe the service to get the runningCount and desiredCount

ecs_client.describe_services(cluster=mycluster,services=[myservice])

  1. while loop if runningCount < desiredCount -- meaning a task is currently being stopped and has not been replaced yet, so don't stop the next task yet!

while myservice['services'][0]['runningCount'] < myservice['services'][0]['desiredCount']:

If the while loop is not true anymore -- meaning both running and desired counts are equal, stop the next task in the list.

This is the actual flow and I am unable to show the actual code as I'm still employed by my current job and all my code belong to them :)

0

If you are using cdk, what I do is change an environment variable (with current date for example) and deploy. This will trigger replacement.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .