The snapshots for one of our AWS volumes are corrupt. We use these snapshots as backup, and in the past they have been a great help. (NB: This is not our only backup method!) A corrupt snapshot is useless however.
I wonder how to handle this, how to detect this beforehand, etc.
Situation
We have an AWS webserver with one large ext3 volume (DATA) with many images in one folder. We make daily snapshots of all volumes, and as we keep them for four weeks, this one is too costly. I just need one snapshot of the images in case of emergency, and for the rest of the volume I want the normal amount. This is what I wanted to do:
- Create snapshot from volume DATA
- Create new ext4 volume IMAGES from snapshot
- Mount volume IMAGES, remove all files and folders except the images folder
- Move the original folder to the root of the volume DATA
- Symlink to the new images folder on IMAGES from the original location on DATA
- Rsync all other data to a new smaller ext4 volume: WEBSITE
- Replace the DATA volume with the WEBSITE volume, linking to the IMAGES volume
Step 3 didn't work. I got the following error:
sudo mount /dev/xvdf /images
mount: mount /dev/xvdf on /images failed: Structure needs cleaning
Googling for this error I found the advice to do an xfs_check, but the filesystem is ext3, so I tried e2fsck. This resulted in endless errors and fixes that didn't seem to work.
sudo xfs_check /dev/xvdf
sudo e2fsck -f /dev/xvdf
I created a new volume, IMAGES, and used rsync to copy everything over, as cp resulted in a crash. I immediately created a snapshot of the new volume, and restored that to see if that worked OK, which it did.
Then I proceeded with splitting the volume, and replace the old volume with the two new ones. This all works, and the problems are solved.
Amazon Support
Still I want to know what happened here, and how to prevent this in the future, so I contacted Amazon Support. They told me that the snapshots were corrupt probably because the snapshots were taken while the volume was in use. We do that all the time, have done many restores with those snapshots (but not this volume), never a problem. This volume was attached, but without writes to it at the time of the snapshot.
I decided to take the advice, detach the volume, make a snapshot, and see what happened. After detaching, the original DATA volume could not be attached anymore. As I had replaced this volume already, it has no consequence, so it's not a big problem, but clearly this doesn't work like adv(ert)ised.
The snapshot can be attached and mounted, and I can open open folders etc. When I perform an e2fsck, I get the errors again. Looking back I forgot to do this e2fsck on the original DATA volume, which is a pity. I guess that would have reported the errors as well.
Amazon Support was below average this time, and that's a pity.
Questions
- How can I detect problems like these without having to test each volume/snapshot manually from time to time?
- Can I set a volume temporarily to write only? How do I do that?
- I read about the
badblocks
command for problems like these (Structure needs cleaning). As I restore a snapshot to a new (virtual) volume, checking that volume seems useless as it's on a different physical location. Is badblocks useful in a case like this? - Fsck seems to change the disk content. What is a safe method to test a problematic disk like this one?