1

I am a bit lost in what exactly happened and how to proceed with a recently expanded zfs configuration on Ubuntu 18.04.

I have a storage server running smoothly for years using ZFS with 2 pools each containing 10+ drives. All was well until.... we decided to expand one pool by adding a new vdev of 10 disks. After plugging it all worked fine. This is what I did to add the devices (Which I now know I should have done on disk-by-id :-( ):

~$ sudo modprobe zfs
~$ dmesg|grep ZFS
[   17.948569] ZFS: Loaded module v0.6.5.6-0ubuntu26, ZFS pool version 5000, ZFS filesystem version 5
~$ lsscsi
[0:0:0:0]    disk    HGST     HUS724020ALS640  A1C4  /dev/sda
[0:0:1:0]    disk    HGST     HUS724020ALS640  A1C4  /dev/sdb
[0:0:2:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdc
[0:0:3:0]    enclosu LSI      SAS2X28          0e12  -
[1:0:0:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdd
[1:0:1:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sde
[1:0:2:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdf
[1:0:3:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdg
[1:0:4:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdh
[1:0:5:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdi
[1:0:6:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdj
[1:0:7:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdk
[1:0:8:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdl
[1:0:9:0]    disk    HGST     HUS726040AL5210  A7J0  /dev/sdm
[1:0:10:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdn
[1:0:11:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdo
[1:0:12:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdp
[1:0:13:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdq
[1:0:14:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdr
[1:0:15:0]   disk    HGST     HUS726060AL5210  A519  /dev/sds
[1:0:16:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdt
[1:0:17:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdu
[1:0:18:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdv
[1:0:19:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdw
[1:0:20:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdx
[1:0:21:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdy
[1:0:22:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdz
[1:0:23:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdaa
[1:0:24:0]   enclosu LSI CORP SAS2X36          0717  -
[1:0:25:0]   disk    HGST     HUS726040AL5210  A7J0  /dev/sdab
[1:0:26:0]   enclosu LSI CORP SAS2X36          0717  -
[1:0:27:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdac      ===>from here below the new plugged disks
[1:0:28:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdad
[1:0:30:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdae
[1:0:31:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdaf
[1:0:32:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdag
[1:0:33:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdah
[1:0:34:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdai
[1:0:35:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdaj
[1:0:36:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdak
[1:0:37:0]   disk    HGST     HUH721010AL4200  A384  /dev/sdal

Next I added the drives as a new raidz2 vdev to the existing archive pool. Seems to be running smoothly afterwards:

~$ sudo zpool add -f archive raidz2 sdac sdad sdae sdaf sdag sdah sdai sdaj sdak sdal
~$ sudo zpool status
  pool: archive
state: ONLINE
  scan: scrub repaired 0 in 17h18m with 0 errors on Sun Dec  8 17:42:17 2019
config:
        NAME                        STATE     READ WRITE CKSUM
        archive                     ONLINE       0     0     0
          raidz2-0                  ONLINE       0     0     0
            scsi-35000cca24311c340  ONLINE       0     0     0
            scsi-35000cca24311ecbc  ONLINE       0     0     0
            scsi-35000cca24d019248  ONLINE       0     0     0
            scsi-35000cca24311e30c  ONLINE       0     0     0
            scsi-35000cca243113ab0  ONLINE       0     0     0
            scsi-35000cca24311c188  ONLINE       0     0     0
            scsi-35000cca24311e7c8  ONLINE       0     0     0
            scsi-35000cca24311e3f0  ONLINE       0     0     0
            scsi-35000cca24311e7bc  ONLINE       0     0     0
            scsi-35000cca24311e40c  ONLINE       0     0     0
            scsi-35000cca243118054  ONLINE       0     0     0
            scsi-35000cca243115cb8  ONLINE       0     0     0
          raidz2-1                  ONLINE       0     0     0
            sdac                    ONLINE       0     0     0
            sdad                    ONLINE       0     0     0
            sdae                    ONLINE       0     0     0
            sdaf                    ONLINE       0     0     0
            sdag                    ONLINE       0     0     0
            sdah                    ONLINE       0     0     0
            sdai                    ONLINE       0     0     0
            sdaj                    ONLINE       0     0     0
            sdak                    ONLINE       0     0     0
            sdal                    ONLINE       0     0     0

errors: No known data errors

  pool: scratch
state: ONLINE
  scan: scrub repaired 0 in 9h8m with 0 errors on Sun Dec  8 09:32:15 2019
config:
        NAME                        STATE     READ WRITE CKSUM
        scratch                     ONLINE       0     0     0
          raidz2-0                  ONLINE       0     0     0
            scsi-35000cca24311e2e8  ONLINE       0     0     0
            scsi-35000cca24311e858  ONLINE       0     0     0
            scsi-35000cca24311ea5c  ONLINE       0     0     0
            scsi-35000cca24311c344  ONLINE       0     0     0
            scsi-35000cca24311e7ec  ONLINE       0     0     0
            scsi-35000cca24311bcb8  ONLINE       0     0     0
            scsi-35000cca24311e8a8  ONLINE       0     0     0
            scsi-35000cca2440b4f98  ONLINE       0     0     0
            scsi-35000cca24311e8f0  ONLINE       0     0     0
            scsi-35000cca2440b4ff0  ONLINE       0     0     0
            scsi-35000cca243113e30  ONLINE       0     0     0
            scsi-35000cca24311e9b4  ONLINE       0     0     0
            scsi-35000cca243137e80  ONLINE       0     0     0

errors: No known data errors

However a reboot most likely messed up the order of disk drives (device assignment; not sure tough but seems most probably). At least that is what I thus far can make of it after reading many many docs and issues. The current status is as below. The scratch pool is working fine. The archive pool not:

~$ sudo zpool status -v
  pool: archive
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid.  There are insufficient replicas for the pool to continue
functioning.
action: Destroy and re-create the pool from
a backup source.
  see: http://zfsonlinux.org/msg/ZFS-8000-5E
  scan: none requested
config:

NAME                        STATE    READ WRITE CKSUM
archive                    UNAVAIL      0    0    0  insufficient replicas
  raidz2-0                  ONLINE      0    0    0
    scsi-35000cca24311c340  ONLINE      0    0    0
    scsi-35000cca24311ecbc  ONLINE      0    0    0
    scsi-35000cca24d019248  ONLINE      0    0    0
    scsi-35000cca24311e30c  ONLINE      0    0    0
    scsi-35000cca243113ab0  ONLINE      0    0    0
    scsi-35000cca24311c188  ONLINE      0    0    0
    scsi-35000cca24311e7c8  ONLINE      0    0    0
    scsi-35000cca24311e3f0  ONLINE      0    0    0
    scsi-35000cca24311e7bc  ONLINE      0    0    0
    scsi-35000cca24311e40c  ONLINE      0    0    0
    scsi-35000cca243118054  ONLINE      0    0    0
    scsi-35000cca243115cb8  ONLINE      0    0    0
  raidz2-1                  UNAVAIL      0    0    0  insufficient replicas
    sdac                    FAULTED      0    0    0  corrupted data
    sdad                    FAULTED      0    0    0  corrupted data
    sdae                    FAULTED      0    0    0  corrupted data
    sdaf                    FAULTED      0    0    0  corrupted data
    sdag                    FAULTED      0    0    0  corrupted data
    sdah                    FAULTED      0    0    0  corrupted data
    sdai                    FAULTED      0    0    0  corrupted data
    sdaj                    FAULTED      0    0    0  corrupted data
    sdak                    FAULTED      0    0    0  corrupted data
    sdal                    FAULTED      0    0    0  corrupted data

  pool: scratch
state: ONLINE
  scan: scrub repaired 0 in 16h36m with 0 errors on Sun Feb  9 17:00:25 2020
config:

NAME                        STATE    READ WRITE CKSUM
scratch                    ONLINE      0    0    0
  raidz2-0                  ONLINE      0    0    0
    scsi-35000cca24311e2e8  ONLINE      0    0    0
    scsi-35000cca24311e858  ONLINE      0    0    0
    scsi-35000cca24311ea5c  ONLINE      0    0    0
    scsi-35000cca24311c344  ONLINE      0    0    0
    scsi-35000cca24311e7ec  ONLINE      0    0    0
    scsi-35000cca24311bcb8  ONLINE      0    0    0
    scsi-35000cca24311e8a8  ONLINE      0    0    0
    scsi-35000cca2440b4f98  ONLINE      0    0    0
    scsi-35000cca24311e8f0  ONLINE      0    0    0
    scsi-35000cca2440b4ff0  ONLINE      0    0    0
    scsi-35000cca243113e30  ONLINE      0    0    0
    scsi-35000cca24311e9b4  ONLINE      0    0    0
    scsi-35000cca243137e80  ONLINE      0    0    0

errors: No known data errors

I tried zpool export archive (also with -f) but it complains of a device missing.

~$ sudo zpool export -f archive
cannot export 'archive': one or more devices is currently unavailable

Obviously importing also fails....

What else to try? I simply cannot believe that a "simple" disk re-ordering messed up all the data on the archive pool.

EDIT March 23rd

The problem is indeed that the drive order has changed.
If I run zdb on the pool it shows me all the info stored in the labels and the large new disks are referred to by wrong /dev/sdxx devices. I determined this by listing the guid of the drives with the actually assigned /dev/sdxx devices and their IDs. It give me the mapping below:

Mapping table of old devs and current devs

But how to fix this. In theory a rewrite of corrected zdb data to the disks should solve this.

0

Ok I am happy again. I was able to solve/fix the issue of reshuffled disks! Posting this answer as reference for someone in the same boat.

Please note this is HIGH RISK work and only for the non faint hearted! Follow these instructions at your own risk and prepare for complete failure of the SYSTEM!

In short how I fixed it for OUR situation;

1) retrieve the ORIGINAL drive path layout of the faulty pool (zdb)

2) Make a ORIGINAL and CURRENT disk/partitionID to path mapping i.e. by fdisk listing all partitions and devices.

3a) mv /dev/sdxx devices and partitions to a TEMPORARY range outside ORIGINAL (at 1)

3b) mv the devices from TEMPORARY range to the ORIGINAL layout

4) pools are recognised (only UNTIL a reboot!) and you can move/copy your data.

5) after rescueing the data I removed all disks from pool en destroyed that pool. Rebuild the pool only AFTER a reboot (note the moved device names).

I'll post some details per point below (all using sudo or as su);

1) zdb This return a long dump of the zdb drive and partition labels per pool. Find for the childs in the affected faulty pool the pair of guid and path. In my case an example:

guid: 16862548186473937209
path: '/dev/sdac1'

2) Create a mapping list of CURRENT and ORIGINAL IDs to paths. This allows the renaming of current device/partition paths to the original layout (of non other original devices are currently used by another new drive not present in the faulty pool!) See my mapping in the update of my question above! link

3) move/rename the devices; Example first CURRENT names to high TEMPORARY range and then from TEMPORARY range to the ORIGINAL layout. I made a bash script to process it quick and allowing double checking and semi-automated generation of the "script". Example;

#!/bin/bash

# move CURRENT TO TEMPORARY
mv /dev/sdac /dev/sdap
mv /dev/sdad /dev/sdaq
mv /dev/sdae /dev/sdar
mv /dev/sdaf /dev/sdas
mv /dev/sdag /dev/sdat
mv /dev/sdah /dev/sdau
mv /dev/sdai /dev/sdav
mv /dev/sdaj /dev/sdaw
mv /dev/sdak /dev/sdax
mv /dev/sdal /dev/sday
mv /dev/sdac1 /dev/sdap1
mv /dev/sdad1 /dev/sdaq1
mv /dev/sdae1 /dev/sdar1
mv /dev/sdaf1 /dev/sdas1
mv /dev/sdag1 /dev/sdat1
mv /dev/sdah1 /dev/sdau1
mv /dev/sdai1 /dev/sdav1
mv /dev/sdaj1 /dev/sdaw1
mv /dev/sdak1 /dev/sdax1
mv /dev/sdal1 /dev/sday1
mv /dev/sdac9 /dev/sdap9
mv /dev/sdad9 /dev/sdaq9
mv /dev/sdae9 /dev/sdar9
mv /dev/sdaf9 /dev/sdas9
mv /dev/sdag9 /dev/sdat9
mv /dev/sdah9 /dev/sdau9
mv /dev/sdai9 /dev/sdav9
mv /dev/sdaj9 /dev/sdaw9
mv /dev/sdak9 /dev/sdax9
mv /dev/sdal9 /dev/sday9

#final move TEMPORARY to ORIGINAL = new CURRENT
mv /dev/sdap /dev/sdai
mv /dev/sdaq /dev/sdaj
mv /dev/sdar /dev/sdak
mv /dev/sdas /dev/sdal
mv /dev/sdat /dev/sdah
mv /dev/sdau /dev/sdag
mv /dev/sdav /dev/sdaf
mv /dev/sdaw /dev/sdae
mv /dev/sdax /dev/sdad
mv /dev/sday /dev/sdac
mv /dev/sdap1 /dev/sdai1
mv /dev/sdaq1 /dev/sdaj1
mv /dev/sdar1 /dev/sdak1
mv /dev/sdas1 /dev/sdal1
mv /dev/sdat1 /dev/sdah1
mv /dev/sdau1 /dev/sdag1
mv /dev/sdav1 /dev/sdaf1
mv /dev/sdaw1 /dev/sdae1
mv /dev/sdax1 /dev/sdad1
mv /dev/sday1 /dev/sdac1
mv /dev/sdap9 /dev/sdai9
mv /dev/sdaq9 /dev/sdaj9
mv /dev/sdar9 /dev/sdak9
mv /dev/sdas9 /dev/sdal9
mv /dev/sdat9 /dev/sdah9
mv /dev/sdau9 /dev/sdag9
mv /dev/sdav9 /dev/sdaf9
mv /dev/sdaw9 /dev/sdae9
mv /dev/sdax9 /dev/sdad9
mv /dev/sday9 /dev/sdac9

4 and 5) After rescue of the data continue with rebuilding. There are plenty of tools and good tutorials that demonstrate best practice to export the pool and to destruct and rebuild it (be sure to rebuild it using disks by identifier and not path :-D ).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.