On FreeBSD 10.2-RELEASE, I had the following setup on a dev (luckily) environment until yesterday:
NAME STATE READ WRITE CKSUM
bigpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
multipath/faba_lun0 ONLINE 0 0 0
multipath/fabb_lun0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
multipath/faba_lun1 ONLINE 0 0 0
multipath/fabb_lun1 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
multipath/faba_lun2 ONLINE 0 0 0
multipath/fabb_lun2 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
multipath/faba_lun3 ONLINE 0 0 0
multipath/fabb_lun3 ONLINE 0 0 0
I then thought it would be a good idea to try and "disable" the 4th mirror (mirror-3) as some sort of rudimentary HA test.
zpool detach bigpool multipath/faba_lun3
Then thinking this was a Bad Idea, I re-added the disk:
zpool add bigpool multipath/faba_lun3
I now have this setup:
NAME STATE READ WRITE CKSUM
bigpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
multipath/faba_lun0 ONLINE 0 0 0
multipath/fabb_lun0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
multipath/faba_lun1 ONLINE 0 0 0
multipath/fabb_lun1 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
multipath/faba_lun2 ONLINE 0 0 0
multipath/fabb_lun2 ONLINE 0 0 0
multipath/faba_lun3 ONLINE 0 0 0
multipath/fabb_lun3 ONLINE 0 0 0
I am unable to re-attach the two lun3's into mirror-3:
root# zpool attach bigpool multipath/faba_lun3 multipath/fabb_lun3
invalid vdev specification
use '-f' to override the following errors:
/dev/multipath/fabb_lun3 is part of active pool 'bigpool'
Seem to be in a catch-22 - Unable to mirror because both vdevs are already part of the pool (therefore an attach doesn't make sense), and unable to remove the vdevs because they're not log, spare, etc...
Does anyone know of a way to "re-enable" this mirror, with existing pool vdevs? I guess this is akin to an online relayout.
Thank you
sc.