I have been using AWS spot instances with persistance storage; something along these lines. I have been this successfuly for months with different instance types, and extended my volume several times as i got more data (128gb > 256gb > 512gb).

In my last experiment, I extended my ebs volume to 1TB, ran some batch process on a m4.16xlarge machine, and after one day, the second status check failed (instance reachability check); unfortunetly i don't have the system log for that moment now.

I thought it's a generic failure, and launched a new c4.8xlarge spot instance with the same ebs volume; no errors, ssh successful, data there.

Now i'm trying to create a new m4.16xlarge with the same ebs, but it fails; full server log here. I'm not able to mount the volume with this or any other instance type now.

Do you have ideas what might have wrong and how i can remedy this?

  • I suggest you read the log yourself and put the parts that look relevant into your question. No-one wants to read a massive log. – Tim Sep 19 '17 at 20:14

The filesystem on your EBS volume is corrupted. Don't ask me how it became that way.

In the log, you'll see the following:

[   14.022730] EXT4-fs (xvdf1): ext4_check_descriptors: Checksum for group 0 failed (8312!=23362)
[   14.029039] EXT4-fs (xvdf1): group descriptors corrupted!
mount: mount /dev/xvdf1 on /permaroot failed: Structure needs cleaning

You'll need to fix your filesystem before trying to mount it again.

After some Googling, I found this, which you can try. But note that I do not warrant it's validity.


| improve this answer | |
  • Thanks Matt, I managed to fix it by: -creating a new instance and attaching the corrupted volume -running sudo fsck /dev/sdf1 (sdf being the device name for the volume) luckily all my working data is there, and all look good! – Md Oliya Sep 20 '17 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.