Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Obviously, Amazon EC2 servers are still running on physical hardware and therefore can have catastrophic failures. And architecturally, I should be building an app that scales horizontally and works around those failures.

How can I simulate an EC2 instance suddenly breaking down? This should have characteristics of a real equipment failure:

  • Processes don't terminate cleanly.
  • Data in memory is not given a chance to write to disk.
  • Files (e.g., on EBS volumes) are not cleanly closed.
  • Open sockets don't FIN, they just hang.
share|improve this question
    
sudo rm -rf /? –  ceejayoz Apr 22 '13 at 20:26
1  
sudo kill -9 -1? –  cyberx86 Apr 22 '13 at 20:32
3  
Netflix has written extensively on this, and has open-sourced much of the code they use to simulate failures. –  EEAA Apr 22 '13 at 20:35
    
Might be interesting to try: echo 1 > /proc/sys/kernel/sysrq followed by echo b > /proc/sysrq-trigger (force restart). –  cyberx86 Apr 22 '13 at 20:45

1 Answer 1

There are an enormous number of ways a system can fail so you probably can't test for all of them and try to work around them.

Perhaps you should look at it from another perspective: look for the services which are essential then find a way to explicitly kill these at random. This simulates a failure you care about, regardless of the cause.

For example if your instance runs an httpd and an FTP server you can kill these daemons occasionally and make sure you can recover from it. You can even programatically terminate the whole server with the AWS API if you want to.

This will also exercise your monitoring infrastructure if the recovery doesn't work :-)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.