I'm wondering what practices should be used to create secure and resilient Rails and mysql servers on EC2. More specifically, when I say "secure and resilient", I am trying to balance the fact that I need to secure the data (both MySQL data and Rails logs) on block-level secured EBS storage using dm-crypt. In order to ensure security, I'm not willing to store passphrases in cleartext. The issue of course is that if a server is restarted, the encrypted file system can't automount, which means I also can't automatically start mysqld / nginx / passenger / etc.

On the other hand, I'm looking for a resilient solution in the sense that failures (such as temporary loss of storage or the failure of an EC2 instance) can be quickly recovered from through some combination of manual intervention and automation.

Is my only option to create a manually invoked shell script to mount the encrypted EBS volume then start the appropriate server processes? Or is there some more sophisticated key management tool that might be able to automate this process in a secure way?

What have folks done in their production server environments for this type of situation?

  • 2
    What crazy threat model do you have such that the highest risk is someone getting the EBS volume, rather than, say, popping your webapp and siphoning your data live? – womble Aug 5 '11 at 1:46
  • @womble - Fair question. Realistically, it's unlikely. The is a compliance requirement for encrypting data at rest. I won't debate that the threat surface is miniscule - basically the only thing this protects against is someone gaining access to the EBS volume without console access to the server where it is mounted. Other mitigating controls exist within the web application itself to reduce the risk you highlight, but I admit that's the more likely successful attack vector. – Chris Hart Aug 5 '11 at 1:54

If all you're doing this for is a compliance requirement for protecting data at rest, then there's no problem with storing the encryption keys in the AMI -- they're separate from the encrypted volumes (far more than most systems, actually, where the keys are on a separate partition of the same disks that hold the encrypted partition). It's not secure against someone getting into the AMI (running or otherwise), but if it's running then they've got the disk data already, and if it's not running then AWS has questions to answer, and at least the data and keys aren't sitting right next to each other.

If you had a real hardware platform, you could use a secure key store hardware module to store the encryption keys, but in this case you're probably limited to manual methods. I can envisage some sort of Rube Goldberg design involving booting instances making a connection to a remote system that did hold the keys, in effect saying "I'm ready to start, please mount my volumes", but the fragility and general ugliness of the solution makes my skin crawl.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.