2

I rebooted a m5d.xlarge EC2 instance running Ubuntu 16.04.5 LTS (xenial) using shutdown -r.

Upon reboot, various drive mounts defined in /etc/fstab did not line up with the device assignments given to them initially.

This caused various services to fail, which rely upon the drive mount with the assigned name containing specific data associated with that name.

In this case, /data needed to contain what is normally expected to be in /data, and not some other drive mount, like /foo or /bar or /whatever-else.

Running lsblk helped with manually redefining the /etc/fstab file to reflect the new device assignments and bringing services back online.

Problem: My worry is that rebooting will randomly reassign device labels to drive mounts, and this problem could or will occur again when the server requires rebooting.

Question: How do I ensure that drive mounts and their respective device paths are preserved between reboots, without needing manual (re)intervention?


Edit: The c5_m5_checks_script.sh script referenced in the answer provided below returns the following information:

# ./c5_m5_checks_script.sh 
------------------------------------------------

OK     NVMe Module is installed and available on your instance


ERROR  NVMe Module is not loaded in the initramfs image.
        - Please run the following command on your instance to recreate initramfs:
        # sudo update-initramfs -c -k all


OK     ENA Module with version 2.0.3K is installed and available on your instance


OK     fstab file looks fine and does not contain any device names.

------------------------------------------------

I don't understand the last OK statement, as the fstab file contains device names. Or Amazon's use of the term "device names" means something different from the device names in the fstab file, perhaps?

To give a specific and concrete example of the nature of the problem, I have an /etc/fstab that looks like this:

...
/dev/nvme2n1    /staging        ext4    defaults,nofail 0       0
/dev/nvme3n1    /data   ext4    defaults,nofail 0       0
...

However, lsblk shows that these two volumes are mounted to each other's device ID:

# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
...
nvme2n1     259:0    0     1T  0 disk /staging
nvme3n1     259:3    0   512G  0 disk /data

In reality, nvme2n1 is the 1T volume /data, and nvme3n1 is the 512G volume /staging.

Something unknown happens between reboots which reassigns the underlying drives to a permutation of the available device IDs.

The only way I know to fix this is to manually umount these two volumes, edit /etc/fstab, and mount them again.

Bounty question: Is there a permanent fix that would let me use a different and permanent identifier in /etc/fstab, which persists consistently between reboots?

  • What did they change from / to? – Tim Sep 9 '19 at 19:20
  • Root did not change. Only those external volumes added via the AWS console and the command line. – Alex Reynolds Sep 9 '19 at 19:24
  • I'll restate - what were the paths before the change, and what were they after the change. This might help work out why it changed. For example I know that nVME and the new instances gives devices new names, and I have a solution AWS suggested, but that might not be your problem. Please edit your question to be a bit more precise if the existing answer wasn't enough for you. – Tim Sep 10 '19 at 0:40
4

AWS has a tool to convert EBS volume device names in /etc/fstab to their UUID values: c5_m5_checks_script.sh

The script also checks that the NVMe module is loaded in your OS image, but your newer instance type would not have launched if that was not present.

| improve this answer | |
1
+250

Why not using LABEL= or UUID= in /etc/fstab?

You can use blkid to find out corresponding labels and UUIDs. Even more info in man fstab of course:

It's also possible to use PARTUUID= and PARTLABEL=. These partitions identifiers are supported for example for GUID Partition Table (GPT).

See mount(8), blkid(8) or lsblk(8) for more details about device identifiers.

Note that mount(8) uses UUIDs as strings. The string representation of the UUID should be based on lower case characters.

| improve this answer | |
  • If you could clarify or otherwise flesh out your answer, that would be very useful. If you're just asking a general question, maybe add a comment instead of an answer? – Alex Reynolds Dec 3 '19 at 5:25
  • man fstab has even an example: LABEL=t-home2 /home ext4 defaults,auto_da_alloc 0 2 – poige Dec 3 '19 at 5:26
  • 1
    Thanks for the help. – Alex Reynolds Dec 9 '19 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.