I'm wondering what practices should be used to create secure and resilient Rails and mysql servers on EC2. More specifically, when I say "secure and resilient", I am trying to balance the fact that I need to secure the data (both MySQL data and Rails logs) on block-level secured EBS storage using dm-crypt. In order to ensure security, I'm not willing to store passphrases in cleartext. The issue of course is that if a server is restarted, the encrypted file system can't automount, which means I also can't automatically start mysqld / nginx / passenger / etc.
On the other hand, I'm looking for a resilient solution in the sense that failures (such as temporary loss of storage or the failure of an EC2 instance) can be quickly recovered from through some combination of manual intervention and automation.
Is my only option to create a manually invoked shell script to mount the encrypted EBS volume then start the appropriate server processes? Or is there some more sophisticated key management tool that might be able to automate this process in a secure way?
What have folks done in their production server environments for this type of situation?