mount: wrong fs type, bad option, bad superblock on /dev/xvdf1, missing codepage or helper program, or other error

Question

I am Unable to mount more than one EBS volumes in an EC2 instance. I have 3 EBS Volumes which are 'root-volume' leftovers from previously terminated EC2 instances(Named: /dev/xvdf1,/dev/xvdg1, /dev/xvde1). I was successfully able to mount /dev/xvde1 using the following command:

#mount /dev/xvde1 /home/ec2-user/xvde1

But when I repeat this procedure for xvdf1 and xvdg1 I am getting this Error:

ec2-user]# mount -t xfs /dev/xvdf1 /home/ec2-user/xvdf1

mount: wrong fs type, bad option, bad superblock on /dev/xvdf1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.

Supporting Outputs: 1) *ec2-user]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part / xvdf 202:80 0 8G 0 disk └─xvdf1 202:81 0 8G 0 part xvdg 202:96 0 8G 0 disk └─xvdg1 202:97 0 8G 0 part xvde 202:64 0 8G 0 disk └─xvde1 202:65 0 8G 0 part /home/ec2-user/xvde1* -->I was able to mount this one successfully.

2)ec2-user]# blkid

/dev/xvda1: LABEL="/" UUID="f25f5092-0401-4edb-9fac-c57f3c673803" TYPE="ext4" PARTLABEL="Linux" PARTUUID="893c59db-bd86-4d67-b40f-221bc82c14c8" /dev/xvdf1: LABEL="/" UUID="f5bd1ae0-85b5-4686-85ff-ed8deb328c92" TYPE="xfs" PARTLABEL="Linux" PARTUUID="870dbb7e-9386-480b-a946-4d0f7ab5c405" /dev/xvdg1: LABEL="/" UUID="f5bd1ae0-85b5-4686-85ff-ed8deb328c92" TYPE="xfs" PARTLABEL="Linux" PARTUUID="870dbb7e-9386-480b-a946-4d0f7ab5c405" /dev/xvde1: LABEL="/" UUID="f5bd1ae0-85b5-4686-85ff-ed8deb328c92" TYPE="xfs" PARTLABEL="Linux" PARTUUID="870dbb7e-9386-480b-a946-4d0f7ab5c405"

3)ec2-user]# file -s /dev/xvdf1

/dev/xvdf1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

ec2-user]# file -s /dev/xvdg1

/dev/xvdg1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

ec2-user]# file -s /dev/xvde1

/dev/xvde1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

4) ec2-user]# mkfs -t xfs /dev/xvdf1 //Tried formatting xvdf1

mkfs.xfs: No such file or directory

Thanks

Answer 1

If the instances were launched using the same AMI then their root volumes will have been created from the same EBS snapshot, so the problem is likely duplicate XFS UUIDs. The error message from mount isn't very helpful, but you may see errors like this in /var/log/messages or equivalent:

Jan 13 23:30:29 ip-172-31-15-234 kernel: XFS (nvme1n1): Filesystem has duplicate UUID 56282b3b-c1f3-425e-90db-e9e26def629d - can't mount

(This example is from a t3 instance using NVMe storage, but it's not NVMe-specific.)

Every XFS filesystem has a (supposedly) unique ID stored on-disk, which protects you from accidentally mounting the same filesystem multiple times. Because the EBS snapshot/restore process is a block-level copy, any volumes you create from a snapshot will have the same UUID as the source volume so you can only mount one at a time.

You can view the UUID for a volume by attaching it but not mounting it, then running xfs_db to examine the attached disk:

# xfs_db -c uuid /dev/nvme1n1
UUID = 56282b3b-c1f3-425e-90db-e9e26def629d

(EDIT: The blkid command will also show you the UUID, even if the device is mounted.)

To work around the issue, you can either use the XFS-specific nouuid mount option to temporarily ignore the duplicate check, e.g.

# mount -t xfs -o nouuid /dev/nvme1n1 /mnt

or you can use xfs_admin to permanently change the UUID on the volume:

# xfs_admin -U generate /dev/nvme1n1
Clearing log and setting UUID
writing all SBs
new UUID = 1eb81512-3f22-4b79-9a35-f22f29745c60

Answer 2

Used the -f (Force) option with mkfs to reformat the partition,

mkfs -t xfs -f /dev/xvdg1*

Re-executed the command

mount /dev/xvdg1 /home/ec2-user/xvdg1/

Output:

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        483M   84K  483M   1% /dev
tmpfs           493M     0  493M   0% /dev/shm
/dev/xvda1      7.8G  1.1G  6.6G  14% /
/dev/xvde1      8.0G  1.3G  6.8G  16% /home/ec2-user/xvde1
/dev/xvdf1      8.0G   33M  8.0G   1% /home/ec2-user/xvdf1
/dev/xvdg1      8.0G   33M  8.0G   1% /home/ec2-user/xvdg1