Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

edit: this seems to be a problem specifically with the AMI that I am trying to shut down. Other AMIs politely shut down with the "sudo poweroff" command.

The goal is for an EC2 instance to shut itself down. All my instances are Ubuntu 10.10.

I used to be able to execute the command "sudo poweroff -n -f" and the instance would shut down. However, now the instances stop responding and peg their CPUs at 50%.

I've tried every possible combination of shutdown/halt/poweroff and they all either:

  • Do nothing, leaving the instance still running
  • Put the instance into this halted/buys but still "on" state

Additional details / exact scenarios:

  • Run sudo poweroff: I get the normal "The system is going down for power off NOW!" message. Then am returned to the shell. The system does not power off. If I execute "sudo poweroff" again, the system seems to shut down (ssh connections terminated), however the AMI does not shut down and remains in a "running" state with constant 50% CPU usage (as monitored from the AWS Management console.

  • Run sudo shutdown now: I get the normal "The system is going down for maintenance NOW!" message but nothing happens. The system keeps running normally. Tailing the syslog I see that the following messages:

-

Dec 21 16:24:36 ip-10-212-177-30 init: ssh main process (599) terminated with status 255
Dec 21 16:24:36 ip-10-212-177-30 init: rc main process (677) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty4 main process (678) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty5 main process (682) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty2 main process (685) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty6 main process (688) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: cron main process (692) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: irqbalance main process (701) killed by TERM signal
Dec 21 16:24:36 ip-10-212-177-30 init: tty3 main process (686) killed by TERM signal

All log messages after calling "sudo poweroff"

sudo poweroff

==> auth.log <==
Dec 21 21:39:37 ip-10-244-74-146 sudo:   ubuntu : TTY=pts/1 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/sbin/poweroff

==> wtmp <==
02~~~runlevel2.6.35-28-virtualR?N?? 
Broadcast message from ubuntu@ip-10-244-74-146
    (/dev/pts/1) at 21:39 ...

The system is going down for power off NOW!

==> boot.log <==
init: Disconnected from system bus
init: dbus main process (663) killed by TERM signal
hostname: the specified hostname is invalid

system is still running, no change

sudo poweroff

==> wtmp <==
~~~shutdown2.6.35-28-virtual)R?N^?

no more log files, system hangs, can't SSH in, CPU spins at 50%

sudo halt -p --verbose

ubuntu@ip-10-2-22-35:~$ /sbin/runlevel 
N 2

ubuntu@ip-10-2-22-35:~$ sudo halt -p --verbose
Calling shutdown

Broadcast message from ubuntu@ip-10-2-22-35
    (/dev/pts/1) at 17:31 ...

The system is going down for power off NOW!
ubuntu@ip-10-2-22-35:~$ /sbin/runlevel 
2 0

ubuntu@ip-10-2-22-35:~$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.1   2892  1732 ?        Ss   17:21   0:00 /sbin/init
root         2  0.0  0.0      0     0 ?        S    17:21   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    17:21   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    17:21   0:00 [migration/0]
root         5  0.0  0.0      0     0 ?        S    17:21   0:00 [watchdog/0]
root         6  0.0  0.0      0     0 ?        S    17:21   0:00 [events/0]
root         7  0.0  0.0      0     0 ?        S    17:21   0:00 [cpuset]
root         8  0.0  0.0      0     0 ?        S    17:21   0:00 [khelper]
root         9  0.0  0.0      0     0 ?        S    17:21   0:00 [netns]
root        10  0.0  0.0      0     0 ?        S    17:21   0:00 [async/mgr]
root        11  0.0  0.0      0     0 ?        S    17:21   0:00 [pm]
root        12  0.0  0.0      0     0 ?        S    17:21   0:00 [xenwatch]
root        13  0.0  0.0      0     0 ?        S    17:21   0:00 [xenbus]
root        14  0.0  0.0      0     0 ?        S    17:21   0:00 [sync_supers]
root        15  0.0  0.0      0     0 ?        S    17:21   0:00 [bdi-default]
root        16  0.0  0.0      0     0 ?        S    17:21   0:00 [kintegrityd/0]
root        17  0.0  0.0      0     0 ?        S    17:21   0:00 [kblockd/0]
root        18  0.0  0.0      0     0 ?        S    17:21   0:00 [ata_aux]
root        19  0.0  0.0      0     0 ?        S    17:21   0:00 [ata_sff/0]
root        20  0.0  0.0      0     0 ?        S    17:21   0:00 [khubd]
root        21  0.0  0.0      0     0 ?        S    17:21   0:00 [kseriod]
root        22  0.0  0.0      0     0 ?        S    17:21   0:00 [kmmcd]
root        24  0.0  0.0      0     0 ?        S    17:21   0:00 [khungtaskd]
root        25  0.0  0.0      0     0 ?        S    17:21   0:00 [kswapd0]
root        26  0.0  0.0      0     0 ?        SN   17:21   0:00 [ksmd]
root        27  0.0  0.0      0     0 ?        S    17:21   0:00 [aio/0]
root        28  0.0  0.0      0     0 ?        S    17:21   0:00 [ecryptfs-kthrea]
root        29  0.0  0.0      0     0 ?        S    17:21   0:00 [crypto/0]
root        35  0.0  0.0      0     0 ?        S    17:21   0:00 [khvcd]
root        36  0.0  0.0      0     0 ?        S    17:21   0:00 [kstriped]
root        37  0.0  0.0      0     0 ?        S    17:21   0:00 [kmpathd/0]
root        38  0.0  0.0      0     0 ?        S    17:21   0:00 [kmpath_handlerd]
root        39  0.0  0.0      0     0 ?        S    17:21   0:00 [ksnapd]
root        40  0.0  0.0      0     0 ?        S    17:21   0:00 [kondemand/0]
root        41  0.0  0.0      0     0 ?        S    17:21   0:00 [kconservative/0]
root       143  0.0  0.0      0     0 ?        S    17:21   0:00 [xfs_mru_cache]
root       144  0.0  0.0      0     0 ?        S    17:21   0:00 [xfslogd/0]
root       145  0.0  0.0      0     0 ?        S    17:21   0:00 [xfsdatad/0]
root       146  0.0  0.0      0     0 ?        S    17:21   0:00 [xfsconvertd/0]
root       148  0.0  0.0      0     0 ?        S    17:21   0:00 [xfsbufd/sda1]
root       149  0.0  0.0      0     0 ?        S    17:21   0:00 [xfsaild/sda1]
root       150  0.0  0.0      0     0 ?        S    17:21   0:00 [xfssyncd/sda1]
root       177  0.0  0.0      0     0 ?        S    17:21   0:00 [flush-202:1]
root       183  0.0  0.0   2560   988 ?        S    17:21   0:00 /sbin/plymouthd --mode=boot --attach-to-session
root       187  0.0  0.0   4484  1260 ?        S    17:21   0:00 mountall --daemon
daemon     382  0.0  0.0   1968   536 ?        Ss   17:22   0:00 portmap
root       413  0.0  0.0   2304   252 ?        Ss   17:22   0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.leases eth0
statd      593  0.0  0.0   2104   824 ?        Ss   17:22   0:00 rpc.statd -L
root       607  0.0  0.0      0     0 ?        S    17:22   0:00 [kjournald]
root       624  0.0  0.0      0     0 ?        S    17:22   0:00 [rpciod/0]
root       627  0.0  0.0      0     0 ?        S<   17:22   0:00 [kslowd000]
root       628  0.0  0.0      0     0 ?        S<   17:22   0:00 [kslowd001]
root       639  0.0  0.0      0     0 ?        S    17:22   0:00 [nfsiod]
ganglia    684  0.0  0.0   4260  1644 ?        Ss   17:22   0:00 /usr/sbin/gmond
root       723  0.0  0.1  52988  3168 ?        Ss   17:22   0:00 php-fpm: master process (/etc/php5/fpm/main.conf)      
www-data   724  0.0  0.1  52972  2536 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   725  0.0  0.1  52972  2540 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   726  0.0  0.1  52972  2540 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   727  0.0  0.1  52972  2540 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   728  0.0  0.1  52972  2540 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   729  0.0  0.1  52972  2540 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   730  0.0  0.1  52972  2540 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   731  0.0  0.1  52972  2540 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   732  0.0  0.1  52972  2540 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   733  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   734  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   735  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   736  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   737  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   738  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   739  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   740  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   741  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   742  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   743  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   744  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   746  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   747  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   748  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   750  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   751  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   753  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   754  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   755  0.0  0.1  52972  2652 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   756  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   757  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   758  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   759  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   760  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   761  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   762  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   763  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   764  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   765  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
www-data   766  0.0  0.1  52972  2656 ?        S    17:22   0:00 php-fpm: pool www                                      
root      1240  0.0  0.1   8848  3116 ?        Ss   17:31   0:00 sshd: ubuntu [priv] 
ubuntu    1334  0.0  0.0   8848  1368 ?        S    17:31   0:00 sshd: ubuntu@pts/1  
ubuntu    1335  0.5  0.3   9016  6048 pts/1    Ss   17:31   0:01 -bash
root      1373  0.0  0.0   1904   512 ?        Ss   17:31   0:00 /bin/sh /etc/init.d/rc 0
root      1375  0.0  0.0   3000  1304 ?        S    17:31   0:00 /bin/bash /etc/rc0.d/K01unregister stop
root      1394  0.0  0.1   5144  1952 ?        S    17:31   0:00 ssh -i /XXXX ubuntu@XXX.com sudo sh -c 'sed -e "/10.2.22.35      fb?"-//W3C//DTD??lang="e
ubuntu    1400  0.0  0.0   4280  1088 pts/1    R+   17:34   0:00 ps aux

last VMS console output after two calls to halt -p (in wierd 50% cpu hung state)

init: tty2 main process (679) killed by TERM signal

init: tty3 main process (680) killed by TERM signal

init: tty6 main process (682) killed by TERM signal

init: hwclock-save main process (881) terminated with status 1

init: Disconnected from system bus

init: dbus main process (618) killed by TERM signal

hostname: the specified hostname is invalid
[   52.346590] System halted.

Thanks!

share|improve this question
    
Are you using a public AMI? If so, what's the ID? –  Mark Johnson Dec 21 '11 at 2:59
    
I am not using a public AMI, however the AMI that I am using was at one time based on a public one. Is there a way to trace it's origin? –  Mike Dec 21 '11 at 3:14
    
Poweroff does still work without issue on Amazon instances - including Ubuntu (just used it 2 days ago). I might suggest checking if you have enabled termination protection (although, it does normally display a message). Trying poweroff without the flags may help to reveal any errors that are logged. Are you able to reboot? (Also, any success with calling shutdown directly)? –  cyberx86 Dec 21 '11 at 12:43
    
Checked and termination protection is off. Can you clarify what you mean by "shutdown directly"? Just "sudo shutdown now"? –  Mike Dec 21 '11 at 16:12
1  
Mike: Did you ever track down what process was hanging the shutdown? –  Eric Hammond Jan 2 '12 at 8:35
show 7 more comments

3 Answers

By default, when you shutdown/halt an EBS boot instance it is simply moved to the "stopped" state. This leave the instance available for restarting. You are no longer paying for instance hours, but you will continue paying a minimal charge for the EBS volume(s).

You can change this behavior so that a shutdown/halt will terminate the instance by setting the instance-initiated-shutdown-behavior to "terminate".

I've written a bit about this here:

Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data
http://alestic.com/2010/01/ec2-instance-locking

You'll want to run the instance with the option:

--instance-initiated-shutdown-behavior terminate

With this flag set, either of the following commands will terminate the EC2 instance:

sudo shutdown -h now
# or
sudo halt

You can also "Change Shutdown Behavior" through the AWS console for running instances (right mouse click).

share|improve this answer
    
Actually the instances aren't even getting to the "stopped" state. They are left in a "started" state but cannot be connected to and the CPUs are pegged at 50%. Already am setting the terminate shutdown behavior and have tried both "sudo shutdown -h now" and "sudo halt" –  Mike Dec 22 '11 at 0:31
add comment

Have you tried:

echo o > /proc/sysrq-trigger

Depending on the kernel that should cause an immediate poweroff. There is no graceful shutdown in this case but if you dont need the image again and start from a fresh one each time, that wont matter.

Some refernce material here: http://en.wikipedia.org/wiki/Magic_SysRq_key#Command_line_access

share|improve this answer
    
This gives me a "Permission denied" error: $ sudo echo o > /proc/sysrq-trigger -bash: /proc/sysrq-trigger: Permission denied –  Mike Dec 21 '11 at 16:23
    
You may need to first enable sysrq: sudo echo 1 > /proc/sys/kernel/sysrq –  cyberx86 Dec 21 '11 at 16:46
    
Same permission denied –  Mike Dec 21 '11 at 21:35
1  
Mike: The "Permission denied" was because the redirect (>) happens before the "sudo". If this command did what you wanted, the correct syntax on Ubuntu would be: echo o | sudo tee /proc/sysrq-trigger –  Eric Hammond Dec 22 '11 at 0:08
1  
How do you know cpu if system is unreachable? –  Eric Hammond Dec 22 '11 at 8:31
show 4 more comments

Look into ec2stop / ec2-stop-instances commands that are a part of the AWS tools

share|improve this answer
    
Those seem to be for stopping a remote instance and require require a Private Key / etc. I want a way to be able to stop the local EC2 instance. –  Mike Dec 21 '11 at 3:38
    
Well, yes and no - it is the interface with AWS via their API to turn off instance - it's somewhat inconvenient that each instance isn't automatically setup to do it for itself, but you can set it up to behave like that by using the appropriate environment variables. –  thinice Dec 21 '11 at 5:55
    
can you expand a bit on how I could use that locally on an instance? My main concern is that I don't want to put any .pem or AWS Identity pairs on my AMIs. –  Mike Dec 21 '11 at 16:03
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.