0

My instance was running for years and suddenly stopped responding Jun 1st. I tried to reboot it, but it would not boot. It gave errors in the system log: https://pastebin.com/rSxr1kLs

Linux version 2.6.32-642.11.1.el6.x86_64 (mockbuild@c1bm.rdu2.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Fri Nov 18 19:25:05 UTC 2016
Kernel command line: root=/dev/xvde ro LANG=en_US.UTF-8 KEYTABLE=us
VFS: Cannot open root device "xvde" or unknown-block(0,0)
Please append a correct "root=" boot option; here are the available partitions:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I tried to detach the EBS volume and re-attach it as /dev/sda1 according to the documentation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html#FilesystemKernel

However, it gave an error Error attaching volume: Invalid value '/dev/sda1' for unixDevice. Attachment point /dev/sda1 is already in use and I was unable to attach it. I re-attached it as /dev/sda but it still won't boot and it still gives the error in the system log.


I was able to launch a new instance into the exact same availability zone and attached my EBS volume as /dev/sdf. It shows up inside the instance as /dev/xvdj. I mounted it with mount /dev/xvdj /xvdj. I can see the grub.conf file:

[root@ip-172-31-4-249 grub]# cat /xvdj/boot/grub/grub.conf
default=0
timeout=1

title CentOS (2.6.32-642.11.1.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-642.11.1.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
title CentOS (2.6.32-504.30.3.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-504.30.3.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-504.30.3.el6.x86_64.img
title CentOS (2.6.32-504.3.3.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-504.3.3.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-504.3.3.el6.x86_64.img
title CentOS (2.6.32-504.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-504.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-504.el6.x86_64.img
title CentOS (2.6.32-431.29.2.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-431.29.2.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-431.29.2.el6.x86_64.img
title CentOS (2.6.32-431.23.3.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-431.23.3.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-431.23.3.el6.x86_64.img

This compares to the grub.conf of the running instance:

[root@ip-172-31-4-249 grub]# cat /boot/grub/grub.conf
default=0
timeout=1

title CentOS-6-x86_64-20130527-03 2.6.32-358.6.2.el6.x86_64
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-358.6.2.el6.x86_64 root=/dev/xvde ro
        initrd /boot/initramfs-2.6.32-358.6.2.el6.x86_64.img

Does it matter that it doesn't have initrd line in the first option?

I tried to mount the EBS volume to the new instance with /dev/sda, but it still wouldn't boot with the same error Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0).

CentOS 6

  • This may or may not be useful: docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html – Chloe Jun 5 '17 at 0:20
  • First choice is to restore the instance from a snapshot and any data from your backups. If you don't have them then I suggest you pay the $30 for developer support (slow) or $100 for business support for a month. Amazon will be able to help you, and will have access to much more information, tools, and experience than us. If your server isn't worth that then start again. – Tim Jun 5 '17 at 0:51
  • Does it matter that it doesn't have initrd line in the first option? – Chloe Jun 5 '17 at 15:18
1

I created a new instance by going to Images > AMIs > Private Images > Selecting the image the instance was started from > Launch. I launched in exactly the same availability zone, not just US or region, but the 2a, 2b, 2c must match as well. I stopped the new instance. I disconnected the EBS volume from the old instance. I re-attached the EBS volume to the new instance at /dev/sdf. I started the new instance. The EBS volume shows up inside the instance as /dev/xvdj so I mounted it with mkdir /xvdj; mount /dev/xvdj /xvdj. I edited /xvdj/boot/grub/grub.conf and changed default=0 to default=1. I saved the file, stopped the new instance, re-attached the EBS volume to the old instance and it started. I ran yum update in the old instance and double-checked /boot/grub/grub.conf and double-checked that it would reboot.

I also found this regarding updates to CentOS kernel: grub.conf missing initrd path after kernel update

I noticed after I ran yum update I now had 2 entries in grub.conf without initrd. Running # yum reinstall kernel.x86_64 works to fix that.

  • 1
    This is what Jason recommended above. You should accept that answer. – EEAA Jun 5 '17 at 15:59
  • Not really, he suggested copying all the data into a new instance, without fixing the actual problem. That would have required a lot of work to set up and configure everything again including file permissions. I do back up the configuration in /etc/ but it would still be a pain, especially with SELinux. Also I would have to set up the security groups, inbound rules, elastic IP, and possible more stuff. Not only that, but it doesn't include enough details, like the fact that you have to use the exact same AMI that you started with, and it has to be in the exact same availability zone + subnet. – Chloe Jun 5 '17 at 16:16
1

I've had this same issue on several occasions and had to solve it by restoring the instance from EBS snapshot backups. Today I had the same issue and was determined to resolve it without having to restore from backups. I did the following:

  1. Detach the root volume from the failed instance /dev/sda1.
  2. Attach the volume into a working instance and mount the volume (e.g. mount /dev/xvdh /xvdhmount)
  3. Back up the boot folder: mv /xvdhmount/boot /xvdhmount/boot-backup
  4. From a working instance with the same version of OS in my case RHEL 7.4 copy the entire contents of the /boot folder via SCP or WinScp into /xvdhmount/.
  5. Detach the volume from the working instance and attach back to the failed instance.
  6. Start the failed instance .... the instance did boot and I am able to log in.

I hope this helps!

0

It looks to me like your kernel got upgraded in such a way that it doesn't understand your root filesystem anymore. Your best bet is to create a new node and mount the EBS volume from the old one as a non-root / non boot device, and transfer the critical data over.

  • Can I downgrade the kernel and update again? I ran yum update in the new instance and it jumped to Linux ip-172-31-4-249 2.6.32-696.3.1.el6.x86_64 #1 SMP Tue May 30 19:52:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux and boots OK. – Chloe Jun 5 '17 at 6:15
0

I came across similar problem, and it turns out, AWS EC2 defaults differ for launching instance vs. creating an AMI: hardware virtualization (HVM) is the default choice in first case, but paravirtual (PV) is the default for image creation.

I stumbled upon this when tried to move EC2 instance between availability zones by snapshotting its EBS volume and creating a new AMI, and this discrepancy in settings (which I did not pay attention too) wasted an hour for me.

tl;dr: just choose HVM when launching from a customized AMI and your instance should boot fine.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.