0

I have a requirement to be able to terminate EC2 instances in under a minute.

The current process takes just under 2 minutes per instance because the OS shutdown process takes 60 seconds. I want to speed up terminations considerably, if possible.

Does anyone know of a way to speed up the terminate() function in EC2? Is there a way to "pull the plug" without a shutdown process as other virtualization solutions do?

Background:
In Boto, I call the terminate() function with the wait_until_terminated() function before handling subnet deletion or other follow-up tasks.

But, I am triggering boto from a 3rd party API that times out if processes (like terminations) take longer than a minute. That means every time I terminate, the API returns errors.

I have tried to work with the 3rd party to increase the timeout, but things like terminations are not in their expected use cases, and as of right now, there is no solution from the 3rd party.

I tried a stop(Force=True) and it is a little faster, but still over a minute.

I tried to forcibly remove the EBS volume, but you have to shutdown the instance first, which brings the process over the 1 minute mark.

I tried SSH'ing in to run various shutdown and halt command arguments, but I cannot find an OS command that runs faster than 60 seconds. The running services are already at a minimum, and I cannot speed up the OS shutdown any further.

I'm hoping to find a way to "pull the plug" via AWS. Or some other method to quickly terminate. It seems like terminations require an OS shutdown, which is a little odd to me when I want to torch the instance anyway.

  • 4
    This sounds like an XY problem. What are you trying to accomplish with this setup? – ceejayoz Mar 7 '17 at 15:56
  • 2
    Why is it important that it be deleted so quickly? – EEAA Mar 7 '17 at 16:49
  • 1
    Can you explain the big picture better. Why would you need to dynamically create and destroy subnets so regularly that this is an issue? I don't think you're going to find a reliable way to turn off an EC2 instance in under a minute, I think you need to reconsider and rearchitect. – Tim Mar 7 '17 at 18:01
  • 2
    It is difficult to see the connection between terminating an instance/waiting for a 3rd party API (caller?) to get a response "from AWS" (for what?)/forking/destroying a subnet. By your own admission, this problem is XY. You need to fully explain and justify the actual problem you are trying to solve, rather than your attempted solution, before we can help. Although the question prima facie seems straightforward enough, I am convinced that that you are asking the wrong question or perhaps trying to solve the wrong problem. I am voting to close this question as "unclear what you're asking." – Michael - sqlbot Mar 7 '17 at 23:54
  • 2
    I'm moderately experienced with AWS, I hold the three associate certifications, I'm 90% of the way through studying for architect pro, and 10% through studying for devops pro. Nothing I've read, and I've read a lot, suggests what you want is possible. AWS is big though, and I'm not calling myself an expert by any means. If you want the definitive answer pay the $29 for a month of developer support and ask AWS directly. – Tim Mar 8 '17 at 8:10
5

While I agree this is an XY problem for sure and you should address the problem in another way, there are far faster ways of doing an OS shutdown than using shutdown. There is no reason to wait for Linux to call init scripts and issue TERM and KILL to all processes.

Historically, I believe killall -9 init or a magic SysRq key was the quickest way. However, systemd lists many ways (man systemd), for example:

   SIGRTMIN+13
       Immediately halts the machine.

   SIGRTMIN+14
       Immediately powers off the machine.

You'll probably have to test a few options before finding the one that AWS reacts to fastest, but going from 60 seconds OS shutdown to 1-5 seconds should be simple enough.

  • 2
    Interesting idea. Kill the OS then call the AWS terminate API to terminate the VM. – Tim Mar 10 '17 at 1:10
  • 2
    @Tim if the instance is set to Auto-Terminate on shutdown, it shouldn't be necessary to call the API – Animism Mar 10 '17 at 1:14
  • I wasn't aware it worked like that, but the documentation suggests you're right. I'd be interested to see what happens when someone tries it. docs.aws.amazon.com/AWSEC2/latest/UserGuide/… – Tim Mar 10 '17 at 1:18
  • 1
    After learning something new, that provides a good answer to the question, it makes me sad you didn't upvote it too :( – Iain Mar 10 '17 at 6:43
  • I forgot about the other ways to kill Linux. I'll test these today. – schroeder Mar 10 '17 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.