2

Is mdadm guaranteed (and trusted via experience) to be safe for taking snapshots with only an xfs_freeze? I have encountered vague warnings about mdadm still working in the background, thus making snapshots unsafe without disassembling the RAID first, but I'd rather avoid having to go through the disassembly/reassembly if possible.

The snapshots are block-level via EBS, but I am less concerned about the snapshotting process than I am about the correct data being consistent when the snapshot is taken.

Options besides mdadm are welcome if they make the process more reliable -- we used to use LVM striping, but switched to mdadm specifically because of reports of cross-disk snapshots not being reliable. We are also looking into some combination, such as using mdadm for striping and an LVM layer for snapshotting.

The ideal solution would be one that avoids having to stop the services running off of the RAID (in this case, Mongo) and would be in the original data format so that a new server could attach an array of the restored snapshots and not require additional steps to massage the data into place. (We already have code that can reassemble snapshots into a new server -- we just need to reliably create those snapshots.)

  • Just curious, why RAID over EBS? PIOPS volumes can deliver enough IOPs to satisfy all but the most very high IO applications, and are much much safer to use. – EEAA Nov 25 '13 at 19:12
  • Specifically, because when we do snapshot, a RAID can do it in parallel, thus the impact of snapshotting (which is substantial) lasts a much shorter amount of time. – Jon Nov 25 '13 at 19:15
  • You know, you don't have to wait for the snapshot to complete before resuming IO, right? As soon as the API call returns, you can resume IO to the volume. – EEAA Nov 25 '13 at 19:16
  • We also have discovered that RAIDs get us much better performance than the single-disk does, even with PIOPS. And yes, we return immediately, but there is a large performance hit on a disk that is being snapshotted under the hood. But that's besides the point of the faster snapshots, which means we can clone a server in under an hour rather than wait for the snapshot to finish in 4+. – Jon Nov 25 '13 at 19:17
  • Oh, sure. Yes agreed on the IO latency caused by the snap. – EEAA Nov 25 '13 at 19:18
2

even this question is rather old I want to give a short answer to the question if snapshotting an EBS-RAID is safe. We're working with PIOPS EBS-RAID0 and we do our regular backups of this RAID with the following procedure:

  1. Stopping the service (DB in our case)
  2. fsfreeze the mountpoint of the RAID (we're using ext4 but this should work with all fs which are able to freeze)
  3. Call the EC2-API to snapshot the devices which are part of the RAID
  4. Wait for the callback of the snapshot (you don't need to wait until the snapshot has finalized - the callback from the API is enough)
  5. unfreeze the RAID
  6. Start the service

The whole procedure takes around 1-2 minutes in our case.

We changed our instances and systems very often in the near past and we always used these snapshots to copy the data to our new instances (and rebuild the RAID there) to reduce the sync time between the replicas. We never had any issues with corrupt data - the snapshoting just works fine!

Hope this helps someone who is searching for an answer.

  • Just making sure - you're using mdadm for the RAID, right? – Jon Apr 25 '14 at 18:02
  • Yes we do - Sorry for not mentioning this in the post! – Osterjour May 6 '14 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.