I am a bit lost in what exactly happened and how to proceed with a recently expanded zfs configuration on Ubuntu 18.04.
I have a storage server running smoothly for years using ZFS with 2 pools each containing 10+ drives. All was well until.... we decided to expand one pool by adding a new vdev of 10 disks. After plugging it all worked fine. This is what I did to add the devices (Which I now know I should have done on disk-by-id :-( ):
~$ sudo modprobe zfs
~$ dmesg|grep ZFS
[ 17.948569] ZFS: Loaded module v0.6.5.6-0ubuntu26, ZFS pool version 5000, ZFS filesystem version 5
~$ lsscsi
[0:0:0:0] disk HGST HUS724020ALS640 A1C4 /dev/sda
[0:0:1:0] disk HGST HUS724020ALS640 A1C4 /dev/sdb
[0:0:2:0] disk HGST HUS726040AL5210 A7J0 /dev/sdc
[0:0:3:0] enclosu LSI SAS2X28 0e12 -
[1:0:0:0] disk HGST HUS726040AL5210 A7J0 /dev/sdd
[1:0:1:0] disk HGST HUS726040AL5210 A7J0 /dev/sde
[1:0:2:0] disk HGST HUS726040AL5210 A7J0 /dev/sdf
[1:0:3:0] disk HGST HUS726040AL5210 A7J0 /dev/sdg
[1:0:4:0] disk HGST HUS726040AL5210 A7J0 /dev/sdh
[1:0:5:0] disk HGST HUS726040AL5210 A7J0 /dev/sdi
[1:0:6:0] disk HGST HUS726040AL5210 A7J0 /dev/sdj
[1:0:7:0] disk HGST HUS726040AL5210 A7J0 /dev/sdk
[1:0:8:0] disk HGST HUS726040AL5210 A7J0 /dev/sdl
[1:0:9:0] disk HGST HUS726040AL5210 A7J0 /dev/sdm
[1:0:10:0] disk HGST HUS726040AL5210 A7J0 /dev/sdn
[1:0:11:0] disk HGST HUS726040AL5210 A7J0 /dev/sdo
[1:0:12:0] disk HGST HUS726040AL5210 A7J0 /dev/sdp
[1:0:13:0] disk HGST HUS726040AL5210 A7J0 /dev/sdq
[1:0:14:0] disk HGST HUS726040AL5210 A7J0 /dev/sdr
[1:0:15:0] disk HGST HUS726060AL5210 A519 /dev/sds
[1:0:16:0] disk HGST HUS726040AL5210 A7J0 /dev/sdt
[1:0:17:0] disk HGST HUS726040AL5210 A7J0 /dev/sdu
[1:0:18:0] disk HGST HUS726040AL5210 A7J0 /dev/sdv
[1:0:19:0] disk HGST HUS726040AL5210 A7J0 /dev/sdw
[1:0:20:0] disk HGST HUS726040AL5210 A7J0 /dev/sdx
[1:0:21:0] disk HGST HUS726040AL5210 A7J0 /dev/sdy
[1:0:22:0] disk HGST HUS726040AL5210 A7J0 /dev/sdz
[1:0:23:0] disk HGST HUS726040AL5210 A7J0 /dev/sdaa
[1:0:24:0] enclosu LSI CORP SAS2X36 0717 -
[1:0:25:0] disk HGST HUS726040AL5210 A7J0 /dev/sdab
[1:0:26:0] enclosu LSI CORP SAS2X36 0717 -
[1:0:27:0] disk HGST HUH721010AL4200 A384 /dev/sdac ===>from here below the new plugged disks
[1:0:28:0] disk HGST HUH721010AL4200 A384 /dev/sdad
[1:0:30:0] disk HGST HUH721010AL4200 A384 /dev/sdae
[1:0:31:0] disk HGST HUH721010AL4200 A384 /dev/sdaf
[1:0:32:0] disk HGST HUH721010AL4200 A384 /dev/sdag
[1:0:33:0] disk HGST HUH721010AL4200 A384 /dev/sdah
[1:0:34:0] disk HGST HUH721010AL4200 A384 /dev/sdai
[1:0:35:0] disk HGST HUH721010AL4200 A384 /dev/sdaj
[1:0:36:0] disk HGST HUH721010AL4200 A384 /dev/sdak
[1:0:37:0] disk HGST HUH721010AL4200 A384 /dev/sdal
Next I added the drives as a new raidz2 vdev to the existing archive pool. Seems to be running smoothly afterwards:
~$ sudo zpool add -f archive raidz2 sdac sdad sdae sdaf sdag sdah sdai sdaj sdak sdal
~$ sudo zpool status
pool: archive
state: ONLINE
scan: scrub repaired 0 in 17h18m with 0 errors on Sun Dec 8 17:42:17 2019
config:
NAME STATE READ WRITE CKSUM
archive ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
scsi-35000cca24311c340 ONLINE 0 0 0
scsi-35000cca24311ecbc ONLINE 0 0 0
scsi-35000cca24d019248 ONLINE 0 0 0
scsi-35000cca24311e30c ONLINE 0 0 0
scsi-35000cca243113ab0 ONLINE 0 0 0
scsi-35000cca24311c188 ONLINE 0 0 0
scsi-35000cca24311e7c8 ONLINE 0 0 0
scsi-35000cca24311e3f0 ONLINE 0 0 0
scsi-35000cca24311e7bc ONLINE 0 0 0
scsi-35000cca24311e40c ONLINE 0 0 0
scsi-35000cca243118054 ONLINE 0 0 0
scsi-35000cca243115cb8 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
sdac ONLINE 0 0 0
sdad ONLINE 0 0 0
sdae ONLINE 0 0 0
sdaf ONLINE 0 0 0
sdag ONLINE 0 0 0
sdah ONLINE 0 0 0
sdai ONLINE 0 0 0
sdaj ONLINE 0 0 0
sdak ONLINE 0 0 0
sdal ONLINE 0 0 0
errors: No known data errors
pool: scratch
state: ONLINE
scan: scrub repaired 0 in 9h8m with 0 errors on Sun Dec 8 09:32:15 2019
config:
NAME STATE READ WRITE CKSUM
scratch ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
scsi-35000cca24311e2e8 ONLINE 0 0 0
scsi-35000cca24311e858 ONLINE 0 0 0
scsi-35000cca24311ea5c ONLINE 0 0 0
scsi-35000cca24311c344 ONLINE 0 0 0
scsi-35000cca24311e7ec ONLINE 0 0 0
scsi-35000cca24311bcb8 ONLINE 0 0 0
scsi-35000cca24311e8a8 ONLINE 0 0 0
scsi-35000cca2440b4f98 ONLINE 0 0 0
scsi-35000cca24311e8f0 ONLINE 0 0 0
scsi-35000cca2440b4ff0 ONLINE 0 0 0
scsi-35000cca243113e30 ONLINE 0 0 0
scsi-35000cca24311e9b4 ONLINE 0 0 0
scsi-35000cca243137e80 ONLINE 0 0 0
errors: No known data errors
However a reboot most likely messed up the order of disk drives (device assignment; not sure tough but seems most probably). At least that is what I thus far can make of it after reading many many docs and issues. The current status is as below. The scratch pool is working fine. The archive pool not:
~$ sudo zpool status -v
pool: archive
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool to continue
functioning.
action: Destroy and re-create the pool from
a backup source.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
scan: none requested
config:
NAME STATE READ WRITE CKSUM
archive UNAVAIL 0 0 0 insufficient replicas
raidz2-0 ONLINE 0 0 0
scsi-35000cca24311c340 ONLINE 0 0 0
scsi-35000cca24311ecbc ONLINE 0 0 0
scsi-35000cca24d019248 ONLINE 0 0 0
scsi-35000cca24311e30c ONLINE 0 0 0
scsi-35000cca243113ab0 ONLINE 0 0 0
scsi-35000cca24311c188 ONLINE 0 0 0
scsi-35000cca24311e7c8 ONLINE 0 0 0
scsi-35000cca24311e3f0 ONLINE 0 0 0
scsi-35000cca24311e7bc ONLINE 0 0 0
scsi-35000cca24311e40c ONLINE 0 0 0
scsi-35000cca243118054 ONLINE 0 0 0
scsi-35000cca243115cb8 ONLINE 0 0 0
raidz2-1 UNAVAIL 0 0 0 insufficient replicas
sdac FAULTED 0 0 0 corrupted data
sdad FAULTED 0 0 0 corrupted data
sdae FAULTED 0 0 0 corrupted data
sdaf FAULTED 0 0 0 corrupted data
sdag FAULTED 0 0 0 corrupted data
sdah FAULTED 0 0 0 corrupted data
sdai FAULTED 0 0 0 corrupted data
sdaj FAULTED 0 0 0 corrupted data
sdak FAULTED 0 0 0 corrupted data
sdal FAULTED 0 0 0 corrupted data
pool: scratch
state: ONLINE
scan: scrub repaired 0 in 16h36m with 0 errors on Sun Feb 9 17:00:25 2020
config:
NAME STATE READ WRITE CKSUM
scratch ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
scsi-35000cca24311e2e8 ONLINE 0 0 0
scsi-35000cca24311e858 ONLINE 0 0 0
scsi-35000cca24311ea5c ONLINE 0 0 0
scsi-35000cca24311c344 ONLINE 0 0 0
scsi-35000cca24311e7ec ONLINE 0 0 0
scsi-35000cca24311bcb8 ONLINE 0 0 0
scsi-35000cca24311e8a8 ONLINE 0 0 0
scsi-35000cca2440b4f98 ONLINE 0 0 0
scsi-35000cca24311e8f0 ONLINE 0 0 0
scsi-35000cca2440b4ff0 ONLINE 0 0 0
scsi-35000cca243113e30 ONLINE 0 0 0
scsi-35000cca24311e9b4 ONLINE 0 0 0
scsi-35000cca243137e80 ONLINE 0 0 0
errors: No known data errors
I tried zpool export archive (also with -f) but it complains of a device missing.
~$ sudo zpool export -f archive
cannot export 'archive': one or more devices is currently unavailable
Obviously importing also fails....
What else to try? I simply cannot believe that a "simple" disk re-ordering messed up all the data on the archive pool.
EDIT March 23rd
The problem is indeed that the drive order has changed.
If I run zdb on the pool it shows me all the info stored in the labels and the large new disks are referred to by wrong /dev/sdxx devices. I determined this by listing the guid of the drives with the actually assigned /dev/sdxx devices and their IDs. It give me the mapping below:
Mapping table of old devs and current devs
But how to fix this. In theory a rewrite of corrected zdb data to the disks should solve this.