1

I have a requirement to be able to terminate EC2 instances in under a minute.

The current process takes just under 2 minutes per instance because the OS shutdown process takes 60 seconds. I want to speed up terminations considerably, if possible.

Does anyone know of a way to speed up the terminate() function in EC2? Is there a way to "pull the plug" without a shutdown process as other virtualization solutions do?

Background:
In Boto, I call the terminate() function with the wait_until_terminated() function before handling subnet deletion or other follow-up tasks.

But, I am triggering boto from a 3rd party API that times out if processes (like terminations) take longer than a minute. That means every time I terminate, the API returns errors.

I have tried to work with the 3rd party to increase the timeout, but things like terminations are not in their expected use cases, and as of right now, there is no solution from the 3rd party.

I tried a stop(Force=True) and it is a little faster, but still over a minute.

I tried to forcibly remove the EBS volume, but you have to shutdown the instance first, which brings the process over the 1 minute mark.

I tried SSH'ing in to run various shutdown and halt command arguments, but I cannot find an OS command that runs faster than 60 seconds. The running services are already at a minimum, and I cannot speed up the OS shutdown any further.

I'm hoping to find a way to "pull the plug" via AWS. Or some other method to quickly terminate. It seems like terminations require an OS shutdown, which is a little odd to me when I want to torch the instance anyway.

Improve this question
17
  • 5
    This sounds like an XY problem. What are you trying to accomplish with this setup?
    – ceejayoz
    Mar 7, 2017 at 15:56
  • 2
    Why is it important that it be deleted so quickly?
    – EEAA
    Mar 7, 2017 at 16:49
  • 1
    Can you explain the big picture better. Why would you need to dynamically create and destroy subnets so regularly that this is an issue? I don't think you're going to find a reliable way to turn off an EC2 instance in under a minute, I think you need to reconsider and rearchitect.
    – Tim
    Mar 7, 2017 at 18:01
  • 2
    It is difficult to see the connection between terminating an instance/waiting for a 3rd party API (caller?) to get a response "from AWS" (for what?)/forking/destroying a subnet. By your own admission, this problem is XY. You need to fully explain and justify the actual problem you are trying to solve, rather than your attempted solution, before we can help. Although the question prima facie seems straightforward enough, I am convinced that that you are asking the wrong question or perhaps trying to solve the wrong problem. I am voting to close this question as "unclear what you're asking."
    – Michael - sqlbot
    Mar 7, 2017 at 23:54
  • 2
    I'm moderately experienced with AWS, I hold the three associate certifications, I'm 90% of the way through studying for architect pro, and 10% through studying for devops pro. Nothing I've read, and I've read a lot, suggests what you want is possible. AWS is big though, and I'm not calling myself an expert by any means. If you want the definitive answer pay the $29 for a month of developer support and ask AWS directly.
    – Tim
    Mar 8, 2017 at 8:10

1 Answer 1

Reset to default
6

While I agree this is an XY problem for sure and you should address the problem in another way, there are far faster ways of doing an OS shutdown than using shutdown. There is no reason to wait for Linux to call init scripts and issue TERM and KILL to all processes.

Historically, I believe killall -9 init or a magic SysRq key was the quickest way. However, systemd lists many ways (man systemd), for example:

   SIGRTMIN+13
       Immediately halts the machine.

   SIGRTMIN+14
       Immediately powers off the machine.

You'll probably have to test a few options before finding the one that AWS reacts to fastest, but going from 60 seconds OS shutdown to 1-5 seconds should be simple enough.

Improve this answer
10
  • 2
    Interesting idea. Kill the OS then call the AWS terminate API to terminate the VM.
    – Tim
    Mar 10, 2017 at 1:10
  • 2
    @Tim if the instance is set to Auto-Terminate on shutdown, it shouldn't be necessary to call the API
    – Animism
    Mar 10, 2017 at 1:14
  • I wasn't aware it worked like that, but the documentation suggests you're right. I'd be interested to see what happens when someone tries it. docs.aws.amazon.com/AWSEC2/latest/UserGuide/…
    – Tim
    Mar 10, 2017 at 1:18
  • 2
    After learning something new, that provides a good answer to the question, it makes me sad you didn't upvote it too :(
    – user9517
    Mar 10, 2017 at 6:43
  • 1
    @Tim unfortunately, no. I was unable to find a way to pass the signals to the Amazon Linux OS.
    – schroeder
    Apr 11, 2017 at 6:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .