I have a requirement to provide a highly available MySQL database with encryption-at-rest on a 2.6.32 linux kernel. The 'highly available' part isn't that hard, but the 'with encryption-at-rest' is proving to be a challenge when used in combination with HA.

The key problem comes with mounting the encrypted storage. On all of our other encrypted-at-rest systems there is a command that needs to be run by a human, who is then prompted for the encryption key. This model has a fairly obvious flaw when it comes to a cluster arrangement where services will have to start automatically.

I am currently at a loss over how to provide encryption-at-rest in an HA environment and not also store key passphrases on the same system.

I can see two possible scenarios, either of which will work with my environment but I'm not sure of the details to make them work. Or even if it is possible.

Scenario 1: CLVM & Cluster

  • A volume is shared between my cluster members.
  • This volume is set up roughly:
    • cryptsetup stuff on the physical device
    • LVM stuff on the new crypt-device
  • Cluster Services are set to not join the cluster automatically, relying on a manual intervention.
  • Cluster Services are started via command run by a human, which supplies the decryption key, which in turn activates the CLVM stuff.

This way running nodes have access to the CLVM volume so they can start services when told to by the cluster manager. Reboots of nodes will still require a human, and the crypt passphrase is never kept on disk anywhere.

Scenario 2: DRBD & Cluster

  • A volume is created on each cluster member
  • cryptsetup stuff is run on the physical device
  • drbd is configured on top of the crypted device, to replicate it between each node
  • LVM or a filesystem is placed on top of the drbd volume
  • Cluster Services are set to not join the cluster automatically, relying on a manual intervention.
  • Cluster services are started by a human who provides the decryption key, which in turn makes the LVM (or filesystem) visible-but-not-mounted.

As with the CLVM setup, nodes don't join the cluster until they have visibility into the presumably-shared storage.

The thing is, I'm not sure if either of the above work that way. Both assume it's possible to layer an LVM PV on top of an encrypted volume (e.g. pvcreate /dev/mapper/cryptmysql). This may not be possible.

  • Yes, you can create an LVM PV on a cryptsetup encrypted volume. (Most major Linux distributions do this automatically when you choose "Encrypt system" during installation.) The key here is, how to get the human to input the key. For locally attached storage, EL will automatically prompt for it on the console at boot time. I have no idea how or if prompting for the passphrase functions with clustered storage. – Michael Hampton Jul 31 '12 at 20:59
  • @MichaelHampton For non-root local filesystems it's actually fairly straight forward, it's what I'm doing on the other systems I have. I just haven't tried layering lvm on top of the crypted volume, I've been doing the encryption after doing the LVM stuff. – sysadmin1138 Jul 31 '12 at 21:02
  • Did you have a look into SEDs and the accompanying support implemented in storage controllers? Or do you really need a software solution for some particular reason? – the-wabbit Jul 31 '12 at 21:13
  • @syneticon-dj I do need a software solution to this. I checked my hardware stack pretty thoroughly looking for ways to handle it lower down the stack, but we haven't bought the right stuff for that. – sysadmin1138 Jul 31 '12 at 21:14
  • If I'm reading this right, it should be possible (at least on EL6 and clones) to set up encryption during installation. Just set up your devices and click Encrypt system. Unfortunately I don't have the necessary setup here to confirm this, but I think it should work. – Michael Hampton Jul 31 '12 at 21:16

The main challenge seems to be the human intervention for key entry. There is some help to that: dm-crypt has support for the TPM which might be available with your platform. See this IBM blueprint for configuration details. Also LUKS/cryptsetup supports reading a slot key from file / from stdin. If you can store your key safely somewhere (like on a smartcard), this might be a viable option.

As for your question whether you can have an LVM PV on a dm-crypt volume: you can, you just would need a run of pvscan / vgchange -a -y after slot unlocking. We've run this kind of setup with much older Kernels a couple of years ago. In the end, we've abandoned it in favor of the use of SEDs for applications with data-at-rest encryption requirements due to performance reasons (dm-crypt used to employ a single thread per encrypted device at that time, which led to a CPU bottleneck in our setup).

| improve this answer | |

It turns out this is entirely doable with LVM as syneticon-dj suggested. Since then I've verified that it works on cluster configs. However, doing so is not as easy as you'd think.

Before the cLVM volume groups are visible, it has to be decrypted via cryptsetup luksOpen on the device needing encryption. This by necessity happens after the cluster services have started, so any resources based on it shouldn't be a member of anything critical such as a stonith device.

Setting up the cluster is the same as it always is, but a few differences:

  • It's a good idea to have a second storage device in the cluster that is NOT encrypted, which could be used by stonith (external/sbd type).
  • Create the clvm resource when all nodes have the crypted volume opened via the same cryptsetup command and mapped to the same device-name (e.g. cryptsetup luksOpen /dev/sdd cryptmysql)
  • Create the clustered volume group on the device created by cryptsetup (e.g. vgcreate ClusterMySQLVG /dev/mapper/cryptmysql)
  • Create a simple bash-script to run the decryption the same way every time, this will be used by an operator post-reboot to get the volume accessible.

Because a node requires manual intervention to be eligible for fail-over, it's a good idea to have more than two nodes in the failover cluster.

Once the volume-group and logical volumes are created as normal, install MySQL. At this point the installation runs as per normal clustering installations.

When a node restarts:

  1. Cluster services will start on boot, and the node will join the cluster. However, as it doesn't have visibility to the MySQL volume, it will not be eligible for fail-over.
  2. An operator connects to the node via ssh, and runs the decrypt script created above, which prompts for the decryption key.
  3. The script runs and creates the needed mapping in /dev/mapper, which LVM then picks up.
  4. The LVM service should automatically refresh and see the new volume group metadata.
  5. At this point cluster services will see that the MySQL volume group is available, and the node will be eligible for fail-over.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.