11

We have a task that loads some configuration files from an external data source. After the settings are uploaded we would like to be able to restart all the tasks in a service so that the settings propagate to all instances.

What's the best way to restart all services?

We have a 'workaround' that involves setting the 'number of tasks' to 0 and then back up, but this is definitely not how it's supposed to be done and has downtime.

  • PS: If someone could create the tag amazon-ecs that would be great :) – Dennkster Jul 14 '15 at 14:49
  • Good call on the tag, I added it for ya. – ceejayoz Jul 14 '15 at 14:55
  • Does this document from Amazon explain the workaround that you are currently using? – Matt Jul 14 '15 at 16:03
8

Using the AWS CLI tool:

aws ecs update-service --force-new-deployment --service my-service
5

What you're wanting to do is essentially the same as redeploying the Service.

To redeploy the service without downtime:

  1. Register a new Task Definition based on the current Task Definition (with the same details)
  2. Call UpdateService, associating the existing Service with the new Task Definition.

This should launch new Tasks for the new Task Definition and then kill the old Tasks for the old Task Definition, effectively restarting the tasks without downtime.

See: UpdateService

  • 1
    I needed to do this via the AWS Console, and this is the easiest way—you can manage the whole process manually if you need to. Helpful when you need to quickly relaunch all the tasks and don't have something more robust set up for the process — in the UI, go to the Task definition, create a new revision, update the service, then after a little time all the Tasks are relaunched! – geerlingguy Jul 6 '17 at 22:02
  • 1
    They've added a checkbox to the service update "Force new deployment" which lets you skip step 1 in your process. – Josh Vickery Mar 19 '18 at 14:22
  • The comment about the "Forces new deployment" option was the Accepted Answer for myself. – ecbrodie Jun 15 '18 at 19:55
2

Task as building block of ECS can be stopped by StopTask call. Service is composed of underlying tasks which may be stopped with same API call. Only missing part here is foreach around results from ListTasks call with defined family parameter. I wrote simple Lambda function which can help you with this.

1

this worked for me:

aws ecs list-tasks --cluster <cluster_name> | jq -r ".taskArns[]" | awk '{print "aws ecs stop-task --cluster <cluster_name> --task \""$0"\""}' | sh

the tasks then recreate on the same instances.

if you need new instances, then use this:

aws ecs list-services --cluster <cluster_name> | jq -r ".serviceArns[]" | awk '{print "aws ecs update-service --cluster <cluster_name> --force-new-deployment  --service \""$0"\""}' | sh
  • That second one seems to do something other than start new instances. – user130681 Nov 10 '18 at 7:25
1

I'm expanding on @user326608's answer above (thanks for the insight!).

This will restart ALL TASKS FOR ALL SERVICES FOR A CLUSTER by stopping all of its tasks. Each service will then automatically launch X number of new tasks, where X is the service's desired task count.

#!/bin/bash

index=0
taskArn=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --query "taskArns[${index}]" --output text)

until [ "$taskArn" = "None" ]
do 
  aws ecs stop-task --cluster ${CLUSTER_NAME} --task $taskArn
  ((index++))
  taskArn=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --query "taskArns[${index}]" --output text)
done
  • Note: If you want to restart tasks for a single service, simply force a new deployment as @Ben Whaley has described. – sudo soul Jul 7 at 0:39
0

Based on the Amazon documentation it seems like you should be able to script the operations in question using the UpdateService API calls. There are some code samples available at the previous link, it seems possible that you should be able to adapt. It seems like writing a script to take care of reloading the services using the appropriate task definition after the updates to task configurations would be the most elegant solution to the problem.

There is more documentation about using AWS CLI with ECS which seems like it would be the easiest way to deal with batch scripting the restarting of services.

  • I can work on writing and posting the script/command sequence but I do not currently have access to an AWS account I would be able to use for testing this sort of thing, so it would be a rough draft/starting point since I would not be able to effectively test it... – Matt Jul 14 '15 at 16:16
0

Works great https://github.com/fdfk/ecsServiceRestart

python ecsServiceRestart.py restart --services="app app2" --cluster=test
0

I have been working on this. It would pretty useful to be able to restart one task at a time reliably. The script below is what I am using now. It is pretty cautious. Requires you hit return for each task. There is a command to wait for the service to be stable, but that does not mean that the task is healthy. And I could in put a time delay. But in the end if things are going bad, the script would just slowly kill the app. So...

#!/bin/bash

if [ $# -eq 2 ]
then
    cluster=$1
    service=$2
else
    echo "Usage: $0 <cluster> <service>"
    exit 1
fi

echo
echo "Restarting $cluster $service tasks:"
echo

for task in $(aws ecs list-tasks --cluster $cluster --service-name $service | awk '{print $2}')
do
    echo
    echo -n "Press enter to stop $task"
    read -r
    echo
    echo "stopping $task..."
    aws ecs stop-task --cluster "$cluster" --task "$task"
    echo
    # aws ecs wait services-stable --cluster "$cluster" --services "$service"    done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.