0

Before:

  • separate SSD boot drive
  • RAID6 4 x 2TB -> /dev/md0
  • LVM PV /dev/md0
  • LVM VG built on the PV above
  • LVM 2 data partitions formed in the VG
    • mounted at /home and /mnt/data
  • I upgraded from ubuntu 20.04 to 22.04 (everything was still working after reboots)

The above was running well, but I upgraded the 2TB drives to 3TB, and added a 5th drive, resulting in RAID6 5 x 3TB.

I didn't partition the new drives, I just did a mdadm --fail/mdadm --remove on each old 2TB drive, followed by a mdadm --add for a new 3TB drive, allowing each to sync before proceeding to the next (keeping tabs on cat /proc/mdstat progress).

The mdadm --detail /dev/md0 shows everything is active, all 5 drives in good state. In short, /dev/md0 looks great. It survives reboots just fine.

root@home:~# mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Sat Sep  3 11:02:41 2016
        Raid Level : raid6
        Array Size : 8790409728 (8.19 TiB 9.00 TB)
     Used Dev Size : 2930136576 (2.73 TiB 3.00 TB)
      Raid Devices : 5
     Total Devices : 5
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu Sep 29 10:53:05 2022
             State : clean
    Active Devices : 5
   Working Devices : 5
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : home:0  (local to host home)
              UUID : 73d52bf2:9c18d305:a554e9ae:e67b7fbc
            Events : 210995

    Number   Major   Minor   RaidDevice State
       4       8       48        0      active sync   /dev/sdd
       5       8        0        1      active sync   /dev/sda
       7       8       32        2      active sync   /dev/sdc
       6       8       16        3      active sync   /dev/sdb
       8       8       64        4      active sync   /dev/sde



root@home:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid6 sde[8] sdd[4] sdc[7] sdb[6] sda[5]
      8790409728 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
      bitmap: 0/11 pages [0KB], 131072KB chunk

unused devices: <none>

However, on boot I now get LVM setup problems. If I run LVM commands with -vv verbosity, they do show that the LVM volumegroup is 'seen' - yet for some reason it won't proceed due to WARNING: device /dev/md0 is an md component, not setting device for PV.

root@home:~# pvscan
  WARNING: device /dev/md0 is an md component, not setting device for PV.
  No matching physical volumes found

root@home:~# pvscan -vv
  devices/hints not found in config: defaulting to all
  metadata/record_lvs_history not found in config: defaulting to 0
  global/locking_type not found in config: defaulting to 1
  devices/md_component_checks not found in config: defaulting to auto
  report/output_format not found in config: defaulting to basic
  log/report_command_log not found in config: defaulting to 0
  Locking /run/lock/lvm/P_global RB
  Device /dev/md0 metadata_version is 1.2.
  /dev/loop0: size is 138880 sectors
  /dev/sda: size is 5860533168 sectors
  /dev/md0: size is 17580819456 sectors
  /dev/loop1: size is 98272 sectors
  /dev/loop2: size is 0 sectors
  /dev/loop3: size is 0 sectors
  /dev/loop4: size is 0 sectors
  /dev/loop5: size is 0 sectors
  /dev/loop6: size is 0 sectors
  /dev/loop7: size is 0 sectors
  /dev/sdb: size is 5860533168 sectors
  /dev/sdc: size is 5860533168 sectors
  /dev/sdd: size is 5860533168 sectors
  /dev/sde: size is 5860533168 sectors
  /dev/sdf: size is 117231408 sectors
  /dev/sdf1: size is 117227520 sectors
  /dev/loop0: using cached size 138880 sectors
  /dev/loop0: using cached size 138880 sectors
  /dev/loop0: No lvm label detected
  /dev/sda: using cached size 5860533168 sectors
  /dev/md0: using cached size 17580819456 sectors
  /dev/md0: using cached size 17580819456 sectors
  /dev/md0: lvm2 label detected at sector 1
  /dev/loop1: using cached size 98272 sectors
  /dev/loop1: using cached size 98272 sectors
  /dev/loop1: No lvm label detected
  /dev/sdb: using cached size 5860533168 sectors
  /dev/sdb: using cached size 5860533168 sectors
  /dev/sdc: using cached size 5860533168 sectors
  /dev/sdd: using cached size 5860533168 sectors
  /dev/sde: using cached size 5860533168 sectors
  /dev/sdf: using cached size 117231408 sectors
  /dev/sdf1: using cached size 117227520 sectors
  /dev/sdf1: using cached size 117227520 sectors
  /dev/sdf1: No lvm label detected
  /dev/loop2: using cached size 0 sectors
  /dev/loop3: using cached size 0 sectors
  /dev/loop4: using cached size 0 sectors
  /dev/loop5: using cached size 0 sectors
  /dev/loop6: using cached size 0 sectors
  /dev/loop7: using cached size 0 sectors
  Locking /run/lock/lvm/V_VolGroupArray RB
  /dev/md0: using cached size 17580819456 sectors
  WARNING: device /dev/md0 is an md component, not setting device for PV.
  Unlocking /run/lock/lvm/V_VolGroupArray
  Reading orphan VG #orphans_lvm2.
  No matching physical volumes found
  Unlocking /run/lock/lvm/P_global

root@home:~# vgscan -vv
  devices/hints not found in config: defaulting to all
  metadata/record_lvs_history not found in config: defaulting to 0
  global/locking_type not found in config: defaulting to 1
  devices/md_component_checks not found in config: defaulting to auto
  Locking /run/lock/lvm/P_global RB
  Device /dev/md0 metadata_version is 1.2.
  /dev/loop0: size is 138880 sectors
  /dev/sda: size is 5860533168 sectors
  /dev/md0: size is 17580819456 sectors
  /dev/loop1: size is 98272 sectors
  /dev/loop2: size is 0 sectors
  /dev/loop3: size is 0 sectors
  /dev/loop4: size is 0 sectors
  /dev/loop5: size is 0 sectors
  /dev/loop6: size is 0 sectors
  /dev/loop7: size is 0 sectors
  /dev/sdb: size is 5860533168 sectors
  /dev/sdc: size is 5860533168 sectors
  /dev/sdd: size is 5860533168 sectors
  /dev/sde: size is 5860533168 sectors
  /dev/sdf: size is 117231408 sectors
  /dev/sdf1: size is 117227520 sectors
  /dev/loop0: using cached size 138880 sectors
  /dev/loop0: using cached size 138880 sectors
  /dev/loop0: No lvm label detected
  /dev/sda: using cached size 5860533168 sectors
  /dev/md0: using cached size 17580819456 sectors
  /dev/md0: using cached size 17580819456 sectors
  /dev/md0: lvm2 label detected at sector 1
  /dev/loop1: using cached size 98272 sectors
  /dev/loop1: using cached size 98272 sectors
  /dev/loop1: No lvm label detected
  /dev/sdb: using cached size 5860533168 sectors
  /dev/sdb: using cached size 5860533168 sectors
  /dev/sdc: using cached size 5860533168 sectors
  /dev/sdd: using cached size 5860533168 sectors
  /dev/sde: using cached size 5860533168 sectors
  /dev/sdf: using cached size 117231408 sectors
  /dev/sdf1: using cached size 117227520 sectors
  /dev/sdf1: using cached size 117227520 sectors
  /dev/sdf1: No lvm label detected
  Obtaining the complete list of VGs to process
  report/output_format not found in config: defaulting to basic
  log/report_command_log not found in config: defaulting to 0
  Processing VG VolGroupArray hLJW0y-qc4O-x5oN-sIg0-5Yxi-JR1R-3A0FEV
  Locking /run/lock/lvm/V_VolGroupArray RB
  /dev/md0: using cached size 17580819456 sectors
  WARNING: device /dev/md0 is an md component, not setting device for PV.
  Unlocking /run/lock/lvm/V_VolGroupArray
  Unlocking /run/lock/lvm/P_global

root@home:~# lvscan -vv
  devices/hints not found in config: defaulting to all
  metadata/record_lvs_history not found in config: defaulting to 0
  global/locking_type not found in config: defaulting to 1
  devices/md_component_checks not found in config: defaulting to auto
  report/output_format not found in config: defaulting to basic
  log/report_command_log not found in config: defaulting to 0
  Locking /run/lock/lvm/P_global RB
  Device /dev/md0 metadata_version is 1.2.
  /dev/loop0: size is 138880 sectors
  /dev/sda: size is 5860533168 sectors
  /dev/md0: size is 17580819456 sectors
  /dev/loop1: size is 98272 sectors
  /dev/loop2: size is 0 sectors
  /dev/loop3: size is 0 sectors
  /dev/loop4: size is 0 sectors
  /dev/loop5: size is 0 sectors
  /dev/loop6: size is 0 sectors
  /dev/loop7: size is 0 sectors
  /dev/sdb: size is 5860533168 sectors
  /dev/sdc: size is 5860533168 sectors
  /dev/sdd: size is 5860533168 sectors
  /dev/sde: size is 5860533168 sectors
  /dev/sdf: size is 117231408 sectors
  /dev/sdf1: size is 117227520 sectors
  /dev/loop0: using cached size 138880 sectors
  /dev/loop0: using cached size 138880 sectors
  /dev/loop0: No lvm label detected
  /dev/sda: using cached size 5860533168 sectors
  /dev/md0: using cached size 17580819456 sectors
  /dev/md0: using cached size 17580819456 sectors
  /dev/md0: lvm2 label detected at sector 1
  /dev/loop1: using cached size 98272 sectors
  /dev/loop1: using cached size 98272 sectors
  /dev/loop1: No lvm label detected
  /dev/sdb: using cached size 5860533168 sectors
  /dev/sdb: using cached size 5860533168 sectors
  /dev/sdc: using cached size 5860533168 sectors
  /dev/sdd: using cached size 5860533168 sectors
  /dev/sde: using cached size 5860533168 sectors
  /dev/sdf: using cached size 117231408 sectors
  /dev/sdf1: using cached size 117227520 sectors
  /dev/sdf1: using cached size 117227520 sectors
  /dev/sdf1: No lvm label detected
  Obtaining the complete list of VGs before processing their LVs
  Processing VG VolGroupArray hLJW0y-qc4O-x5oN-sIg0-5Yxi-JR1R-3A0FEV
  Locking /run/lock/lvm/V_VolGroupArray RB
  /dev/md0: using cached size 17580819456 sectors
  WARNING: device /dev/md0 is an md component, not setting device for PV.
  Unlocking /run/lock/lvm/V_VolGroupArray
  Unlocking /run/lock/lvm/P_global

Guide me to provide the appropriate output to help diagnose this, please.

Edit 1:

The following is after I've commented out the LVM partitions in /etc/fstab.

output of lsblk:

root@home:~# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0    7:0    0 67.8M  1 loop  /snap/lxd/22753
loop1    7:1    0   48M  1 loop  /snap/snapd/17029
sda      8:0    0  2.7T  0 disk
└─md0    9:0    0  8.2T  0 raid6
sdb      8:16   0  2.7T  0 disk
└─md0    9:0    0  8.2T  0 raid6
sdc      8:32   0  2.7T  0 disk
└─md0    9:0    0  8.2T  0 raid6
sdd      8:48   0  2.7T  0 disk
└─md0    9:0    0  8.2T  0 raid6
sde      8:64   0  2.7T  0 disk
└─md0    9:0    0  8.2T  0 raid6
sdf      8:80   0 55.9G  0 disk
└─sdf1   8:81   0 55.9G  0 part  /

output of blkid:

root@home:~# blkid
/dev/sdf1: UUID="751e6495-f77b-49a0-b170-2be146064752" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="96cba4d1-01"
/dev/sdd: UUID="73d52bf2-9c18-d305-a554-e9aee67b7fbc" UUID_SUB="eaa0c0ab-08e9-bbd9-79bf-18386fcee952" LABEL="home:0" TYPE="linux_raid_member"
/dev/sdb: UUID="73d52bf2-9c18-d305-a554-e9aee67b7fbc" UUID_SUB="06927a3a-e179-95d0-ae6d-591faed89962" LABEL="home:0" TYPE="linux_raid_member"
/dev/md0: UUID="PjdxdD-aMVt-5qDb-ZtHJ-tzec-K0ho-l92bbW" TYPE="LVM2_member"
/dev/sde: UUID="73d52bf2-9c18-d305-a554-e9aee67b7fbc" UUID_SUB="34a14cf3-3ebe-c11b-c2d1-289c1b6a3527" LABEL="home:0" TYPE="linux_raid_member"
/dev/sdc: UUID="73d52bf2-9c18-d305-a554-e9aee67b7fbc" UUID_SUB="a24fee08-76a9-09cf-2650-963f1c2431b0" LABEL="home:0" TYPE="linux_raid_member"
/dev/sda: UUID="73d52bf2-9c18-d305-a554-e9aee67b7fbc" UUID_SUB="4dbbe2cb-5cd4-22f1-4dee-26d0079c6d1a" LABEL="home:0" TYPE="linux_raid_member"
/dev/loop1: TYPE="squashfs"
/dev/loop0: TYPE="squashfs"

output from mdadm --examine /dev/md0:

root@home:~# mdadm --examine /dev/md0
mdadm: No md superblock detected on /dev/md0.

output from lvmdiskscan:

root@home:~# lvmdiskscan
  /dev/loop0 [      67.81 MiB]
  /dev/md0   [      <8.19 TiB] LVM physical volume
  /dev/loop1 [      47.98 MiB]
  /dev/sdf1  [     <55.90 GiB]
  0 disks
  3 partitions
  0 LVM physical volume whole disks
  1 LVM physical volume
Improve this question
5
  • Could you please also attach lsblk and blkid?
    – Nikita Kipriyanov
    Commented Sep 30, 2022 at 4:55
  • @NikitaKipriyanov - added. (thanks for the response)
    – James Addison
    Commented Sep 30, 2022 at 5:59
  • Wiping any stray MD superblock that may be seen on the /dev/md0 comes to mind, but this might be dangerous. That could be done with mdadm --zero-superblock /dev/md0. Determine which exactly device locations this command zeroes, examine them to check for anything that resembles a MD superblock, backup all of them using dd, also try to backup your LVM metadata, and then try the mentioned command to remove whatever confuses LVM.
    – Nikita Kipriyanov
    Commented Sep 30, 2022 at 6:56
  • Also, try to run mdadm --examine /dev/md0 to see if mdadm itself recognizes it as a component of some MD array. (--examine is used to check component devices, in contrast with --detail, but in your case LVM claims md0 itself is a component device, so we're trying to verify that.)
    – Nikita Kipriyanov
    Commented Sep 30, 2022 at 7:01
  • @NikitaKipriyanov, mdadm --examine /dev/md0 output added.
    – James Addison
    Commented Sep 30, 2022 at 15:15

1 Answer 1

Reset to default
0

What fixed the problem?

To address the issue, I had to modify LVM configuration in /etc/lvm/lvm.conf. Specifically, I had to change:

devices {
        ...

        md_component_detection = 1
}

to

devices {
        ...

        md_component_detection = 0
}

Configuration notes on the md_component_detection setting:

    # Configuration option devices/md_component_detection.
    # Enable detection and exclusion of MD component devices.
    # An MD component device is a block device that MD uses as part
    # of a software RAID virtual device. When an LVM PV is created
    # on an MD device, LVM must only use the top level MD device as
    # the PV, and should ignore the underlying component devices.
    # In cases where the MD superblock is located at the end of the
    # component devices, it is more difficult for LVM to consistently
    # identify an MD component, see the md_component_checks setting.

I had been running with LVM defaults for years; I'd never even looked at the configuration before.

Why did this happen?

I suspect it was because I added the raw 3TB disks to the existing RAID6 array rather than create a partition on each disk and add that to the array. (so I added /dev/sda rather than /dev/sda1). Each raw disk added would have had the MD superblock, and when combined, the placement of those superblocks 'confused' LVM's detection.

It would be good to have confirmation on the above from those more experienced, but the change above is what 'fixed' my problem.

Improve this answer
3
  • 1
    While I know about such a "solution", I think it is dangerous too. The motivation for that MD component detection is enabled by default is that if you have e.g. the RAID1, LVM should not bother to use individual drives, despite the fact technically it could. So it checks if the disk could be a component and if it does, it is not going to use it. I'd check thoroughly and file a bug against LVM, because this looks like a bug (a false positive) in the detection code.
    – Nikita Kipriyanov
    Commented Oct 1, 2022 at 17:30
  • @NikitaKipriyanov where would I submit an issue for LVM? I'm happy to do so!
    – James Addison
    Commented Oct 2, 2022 at 23:09
  • The actual Ubuntu bug tracker is bugs.launchpad.net/ubuntu , but please begin with reading here: help.ubuntu.com/community/ReportingBugs . I feel I need to make a little explanation: although the problem is probably in the LVM tools, you use a distribution built tools. So the distribution maintainers take all the responsibility to support their build. If they discover the bug was not introduced in their build but rather it exists in the upstream, they either file it to upstream themselves or ask you to do that.
    – Nikita Kipriyanov
    Commented Oct 3, 2022 at 5:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .