2

i have a VG that uses the PVs /dev/md1 and /dev/md2. Now one hdd failed in /dev/md1:

[root@xenserver1 ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda3[0]
      382320128 blocks [2/1] [U_]

md2 : active raid1 sdc4[0] sdd4[1]
      1562805632 blocks [2/2] [UU]

md0 : active raid1 sda1[0]
      4194240 blocks [2/1] [U_]

unused devices: <none>

since that, I get the following LVM error:

[root@xenserver1 ~]# pvscan
  WARNING: Volume Group VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca is not consistent
  PV /dev/md1   VG VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca   lvm2 [364.61 GB / 150.25 GB free]
  PV /dev/md2   VG VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca   lvm2 [1.46 TB / 1.35 TB free]
  Total: 2 [1.81 TB] / in use: 2 [1.81 TB] / in no VG: 0 [0   ]

I tried vgcfgrestore --file /etc/lvm/backup/VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca, but without success.

If I execeute vgscan I get:

[root@xenserver1 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Volume group "VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca" inconsistent, run as master to correct
*** glibc detected *** vgscan: corrupted double-linked list: 0x09048780 ***
======= Backtrace: =========
/lib/libc.so.6[0xb75b9629]
/lib/libc.so.6(cfree+0x59)[0xb75b99e9]
/lib/libdevmapper.so.1.02(dm_pool_destroy+0x3b)[0xb779bd5b]
vgscan(vg_release+0x38)[0x80a8dc8]
vgscan[0x806cc2a]
vgscan(process_each_vg+0x12c)[0x806d96c]
vgscan(vgscan+0x13c)[0x80755ec]
vgscan(lvm_run_command+0x11e5)[0x8060175]
vgscan(lvm2_main+0x4ca)[0x806067a]
vgscan(main+0x32)[0x80772d2]
/lib/libc.so.6(__libc_start_main+0xdc)[0xb7565e9c]
vgscan[0x8053421]
======= Memory map: ========
08048000-080f9000 r-xp 00000000 09:00 478343     /usr/sbin/lvm
080f9000-080fe000 rw-p 000b1000 09:00 478343     /usr/sbin/lvm
080fe000-08105000 rw-p 00000000 00:00 0
09003000-09068000 rw-p 00000000 00:00 0          [heap]
b7200000-b7221000 rw-p 00000000 00:00 0
b7221000-b7300000 ---p 00000000 00:00 0
b7336000-b7341000 r-xp 00000000 09:00 327931     /lib/libgcc_s-4.1.2-20080825.so.1
b7341000-b7342000 rw-p 0000a000 09:00 327931     /lib/libgcc_s-4.1.2-20080825.so.1
b7347000-b734e000 r--s 00000000 09:00 410934     /usr/lib/gconv/gconv-modules.cache
b734e000-b754e000 r--p 00000000 09:00 410974     /usr/lib/locale/locale-archive
b754e000-b7550000 rw-p 00000000 00:00 0
b7550000-b76a2000 r-xp 00000000 09:00 327906     /lib/libc-2.5.so
b76a2000-b76a4000 r--p 00152000 09:00 327906     /lib/libc-2.5.so
b76a4000-b76a5000 rw-p 00154000 09:00 327906     /lib/libc-2.5.so
b76a5000-b76a8000 rw-p 00000000 00:00 0
b76a8000-b76e8000 r-xp 00000000 09:00 410540     /usr/lib/libncurses.so.5.5
b76e8000-b76f0000 rw-p 00040000 09:00 410540     /usr/lib/libncurses.so.5.5
b76f0000-b76f1000 rw-p 00000000 00:00 0
b76f1000-b76f4000 r-xp 00000000 09:00 327921     /lib/libdl-2.5.so
b76f4000-b76f5000 r--p 00002000 09:00 327921     /lib/libdl-2.5.so
b76f5000-b76f6000 rw-p 00003000 09:00 327921     /lib/libdl-2.5.so
b76f6000-b7731000 r-xp 00000000 09:00 327957     /lib/libsepol.so.1
b7731000-b7732000 rw-p 0003b000 09:00 327957     /lib/libsepol.so.1
b7732000-b773c000 rw-p 00000000 00:00 0
b773c000-b7752000 r-xp 00000000 09:00 327956     /lib/libselinux.so.1
b7752000-b7754000 rw-p 00015000 09:00 327956     /lib/libselinux.so.1
b7754000-b7783000 r-xp 00000000 09:00 410572     /usr/lib/libreadline.so.5.1
b7783000-b7787000 rw-p 0002f000 09:00 410572     /usr/lib/libreadline.so.5.1
b7787000-b7789000 rw-p 00000000 00:00 0
b7789000-b77a8000 r-xp 00000000 09:00 327920     /lib/libdevmapper.so.1.02
b77a8000-b77aa000 rw-p 0001e000 09:00 327920     /lib/libdevmapper.so.1.02
b77aa000-b77bf000 r-xp 00000000 09:00 327953     /lib/libpthread-2.5.so
b77bf000-b77c0000 r--p 00015000 09:00 327953     /lib/libpthread-2.5.so
b77c0000-b77c1000 rw-p 00016000 09:00 327953     /lib/libpthread-2.5.so
b77c1000-b77c3000 rw-p 00000000 00:00 0
b77c3000-b77c7000 r-xp 00000000 09:00 327918     /lib/libdevmapper-event.so.1.02
b77c7000-b77c8000 rw-p 00003000 09:00 327918     /lib/libdevmapper-event.so.1.02
b77cb000-b77cc000 rw-p 00000000 00:00 0
b77cc000-b77cd000 r--p 00796000 09:00 410974     /usr/lib/locale/locale-archive
b77cd000-b77ce000 rw-p 00000000 00:00 0
b77ce000-b77cf000 r-xp 00000000 00:00 0          [vdso]
b77cf000-b77ea000 r-xp 00000000 09:00 327897     /lib/ld-2.5.so
b77ea000-b77eb000 r--p 0001a000 09:00 327897     /lib/ld-2.5.so
b77eb000-b77ec000 rw-p 0001b000 09:00 327897     /lib/ld-2.5.so
bfd24000-bfd39000 rw-p 00000000 00:00 0          [stack]
Aborted

Is there any possibility to restore the LVs on this VG? What does run as master to correct mean?

EDIT

this ist the output of mdadm -D /dev/md1

[root@xenserver1 ~]# mdadm -D /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Thu Sep 22 23:14:42 2011
     Raid Level : raid1
     Array Size : 382320128 (364.61 GiB 391.50 GB)
  Used Dev Size : 382320128 (364.61 GiB 391.50 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sat Dec  7 01:25:25 2013
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : 3d3c12fd:51ab6cd6:83470809:bba18a55
         Events : 0.49196839

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       0        0        1      removed

EDIT2

Raid /dev/md1 is now clean again

[root@xenserver1 ~]# mdadm -D /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Thu Sep 22 23:14:42 2011
     Raid Level : raid1
     Array Size : 382320128 (364.61 GiB 391.50 GB)
  Used Dev Size : 382320128 (364.61 GiB 391.50 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sat Dec  7 09:14:22 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 3d3c12fd:51ab6cd6:83470809:bba18a55
         Events : 0.49196863

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       67        1      active sync   /dev/sde3

EDIT3

if i want to activate the VG i get a different error message:

[root@xenserver1 ~]# vgchange VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca -ay
  Volume group "VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca" inconsistent, run as master to correct
*** glibc detected *** vgchange: double free or corruption (!prev): 0x0a033468 ***
======= Backtrace: =========
/lib/libc.so.6[0xb74e05a5]
/lib/libc.so.6(cfree+0x59)[0xb74e09e9]
/lib/libdevmapper.so.1.02(dm_pool_destroy+0x3b)[0xb76c2d5b]
vgchange(vg_release+0x38)[0x80a8dc8]
vgchange[0x806cc2a]
vgchange(process_each_vg+0x3b5)[0x806dbf5]
vgchange(vgchange+0x347)[0x806ee37]
vgchange(lvm_run_command+0x11e5)[0x8060175]
vgchange(lvm2_main+0x4ca)[0x806067a]
vgchange(main+0x32)[0x80772d2]
/lib/libc.so.6(__libc_start_main+0xdc)[0xb748ce9c]
vgchange[0x8053421]
======= Memory map: ========
08048000-080f9000 r-xp 00000000 09:00 478343     /usr/sbin/lvm
080f9000-080fe000 rw-p 000b1000 09:00 478343     /usr/sbin/lvm
080fe000-08105000 rw-p 00000000 00:00 0
09ff1000-0a058000 rw-p 00000000 00:00 0          [heap]
b725d000-b7268000 r-xp 00000000 09:00 327931     /lib/libgcc_s-4.1.2-20080825.so.1
b7268000-b7269000 rw-p 0000a000 09:00 327931     /lib/libgcc_s-4.1.2-20080825.so.1
b726e000-b7275000 r--s 00000000 09:00 410934     /usr/lib/gconv/gconv-modules.cache
b7275000-b7475000 r--p 00000000 09:00 410974     /usr/lib/locale/locale-archive
b7475000-b7477000 rw-p 00000000 00:00 0
b7477000-b75c9000 r-xp 00000000 09:00 327906     /lib/libc-2.5.so
b75c9000-b75cb000 r--p 00152000 09:00 327906     /lib/libc-2.5.so
b75cb000-b75cc000 rw-p 00154000 09:00 327906     /lib/libc-2.5.so
b75cc000-b75cf000 rw-p 00000000 00:00 0
b75cf000-b760f000 r-xp 00000000 09:00 410540     /usr/lib/libncurses.so.5.5
b760f000-b7617000 rw-p 00040000 09:00 410540     /usr/lib/libncurses.so.5.5
b7617000-b7618000 rw-p 00000000 00:00 0
b7618000-b761b000 r-xp 00000000 09:00 327921     /lib/libdl-2.5.so
b761b000-b761c000 r--p 00002000 09:00 327921     /lib/libdl-2.5.so
b761c000-b761d000 rw-p 00003000 09:00 327921     /lib/libdl-2.5.so
b761d000-b7658000 r-xp 00000000 09:00 327957     /lib/libsepol.so.1
b7658000-b7659000 rw-p 0003b000 09:00 327957     /lib/libsepol.so.1
b7659000-b7663000 rw-p 00000000 00:00 0
b7663000-b7679000 r-xp 00000000 09:00 327956     /lib/libselinux.so.1
b7679000-b767b000 rw-p 00015000 09:00 327956     /lib/libselinux.so.1
b767b000-b76aa000 r-xp 00000000 09:00 410572     /usr/lib/libreadline.so.5.1
b76aa000-b76ae000 rw-p 0002f000 09:00 410572     /usr/lib/libreadline.so.5.1
b76ae000-b76b0000 rw-p 00000000 00:00 0
b76b0000-b76cf000 r-xp 00000000 09:00 327920     /lib/libdevmapper.so.1.02
b76cf000-b76d1000 rw-p 0001e000 09:00 327920     /lib/libdevmapper.so.1.02
b76d1000-b76e6000 r-xp 00000000 09:00 327953     /lib/libpthread-2.5.so
b76e6000-b76e7000 r--p 00015000 09:00 327953     /lib/libpthread-2.5.so
b76e7000-b76e8000 rw-p 00016000 09:00 327953     /lib/libpthread-2.5.so
b76e8000-b76ea000 rw-p 00000000 00:00 0
b76ea000-b76ee000 r-xp 00000000 09:00 327918     /lib/libdevmapper-event.so.1.02
b76ee000-b76ef000 rw-p 00003000 09:00 327918     /lib/libdevmapper-event.so.1.02
b76f3000-b76f4000 r--p 00796000 09:00 410974     /usr/lib/locale/locale-archive
b76f4000-b76f5000 rw-p 00000000 00:00 0
b76f5000-b76f6000 r-xp 00000000 00:00 0          [vdso]
b76f6000-b7711000 r-xp 00000000 09:00 327897     /lib/ld-2.5.so
b7711000-b7712000 r--p 0001a000 09:00 327897     /lib/ld-2.5.so
b7712000-b7713000 rw-p 0001b000 09:00 327897     /lib/ld-2.5.so
bff17000-bff2c000 rw-p 00000000 00:00 0          [stack]
Aborted
5
  • What the OS? You have to rebuild that raid1 and for md1 to clear the glibc errror most likely after that your LVM VG should be fine again. Dec 7, 2013 at 3:48
  • I've added the mdadm output - shouldn't it be possible to go with a degraded array until i have the new disk?
    – Stephan
    Dec 7, 2013 at 6:21
  • by the way: the OS is Citrix XenServer 5.6.1 (CentOS)
    – Stephan
    Dec 7, 2013 at 7:03
  • ok, I rebuilt the array (it is in clean state now) - but i still get the "WARNING: Volume Group VG_XenStorage-00c0d7a2-7054-c4fc-414a-e6c6a43b9cca is not consistent" message if I do pvscan
    – Stephan
    Dec 7, 2013 at 8:37
  • Hmm, interesting. Can you paste what "vgdisplay" shows? Dec 8, 2013 at 1:39

0

You must log in to answer this question.

Browse other questions tagged .