11

After Amazon's Aug 8 outage, all (EBS based) AMIs stopped working for many users. This is due to corruption of some sectors in snapshots that the AMIs are based on.

However, Amazon created recovery snapshots where the disk problems should be fixed. Those are named along the lines of "Recovery snapshot for vol-xxxxxxxx".

I created a new AMI from recovery snapshot which worked fine, but instances launched from this new AMI do not work: their state is "Running", but I cannot ssh into the machine nor access any of the web services that should be running there. It boils down to this (from System Log, accessible through AWS management console):

EXT3-fs: sda1: couldn't mount because of unsupported optional features (240).

EXT2-fs: sda1: couldn't mount because of unsupported optional features (244).

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

I've mounted a volume created from that recovery snapshot in a another server on AWS, and everything looks quite normal though. For example, fsck says:

$ sudo fsck -a /dev/xvdg
fsck from util-linux-ng 2.17.2
uec-rootfs: clean, 53781/524288 files, 546065/2097152 blocks

In one of the AWS forum discussions, I found this advice from someone with similar problems:

A work around will be to make a volume from the snapshot and attach it to a running instance, use fsck --force to force the checking of the filesystem and once cleared, you can make a snapshot and use it for the AMI.

But I don't know how to force fsck on Ubuntu (11.04):

$ sudo fsck --force /dev/xvdg
fsck from util-linux-ng 2.17.2
fsck.ext3: invalid option -- 'o'

Anyone know how to force file system check on the volume on Ubuntu? Any other ideas on how to launch working instances that are based on the recovery snapshot?

Right now it looks like it might be quicker to just start over from a clean Ubuntu AMI and re-setup all our services. :-( But of course I would prefer not to do that if there's any way to get the recovery snapshot to actually work.

3 Answers 3

14

I ran into the same problem when trying to duplicate a machine.

The problem turned out to be the kernel. Both when creating the AMI and the instance I selected default for the kernel image.

To resolve the problem, I recreated the AMI using the same kernel image as the original instance.

2
  • To clarify, the default kernel image lacks ext4 support, but the kernel that was used to build the AMI should always be used anyway.
    – DCYorke
    Aug 31, 2011 at 21:23
  • If only the snapshot remains, it will be very hard to recover. Can you suggest a method to backup this kind of metadata (also, which security groups and user-data are used) with the snapshot or somewhere else? Jun 24, 2014 at 15:20
2

Could you try the following command (note -f option instead of --force): sudo fsck -f /dev/xvdg

Hope this helps. Fred

1
  • fsck -f indeed does something more (don't know exactly what; man fsck doesn't say anything about it), so +1. But in any case this doesn't solve the whole problem; I created a snapshot and then an AMI from the fscked volume, and lauched an instance from that, and still get the same "Kernel panic ... Unable to mount root" error in System Log.
    – Jonik
    Aug 29, 2011 at 13:05
0

I didn't want to waste more time fighting with weird AWS-specific problems, so I created a new clean instance from one of the official Ubuntu AMIs (in my case ami-359ea941 which is a 32-bit EBS-backed image of Ubuntu 11.04 in the eu-west-1 region), and re-created my server setup there.

The fact that I could mount a volume created from the recovery snapshot in the new instance made the re-setup much faster though. For example, I did something like cp -a /mnt/recovery/usr/local /usr to restore a whole lot of stuff under /usr/local.

So in my case the recovery backups were far from useless as I could access the data on them. But of course it would still have been nicer to just create an AMI from the snapshot and continue using (instances from) that like the whole incident never happened. (Feel free to add an answer if you know how to achieve that!)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .