Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a zpool with 4 2TB USB disks in a raidz config:

[root@chef /mnt/Chef]# zpool status farcryz1
  pool: farcryz1
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    farcryz1    ONLINE       0     0     0
      raidz1    ONLINE       0     0     0
        da1     ONLINE       0     0     0
        da2     ONLINE       0     0     0
        da3     ONLINE       0     0     0
        da4     ONLINE       0     0     0

In order to test the pool, I simulated a drive failure by pulling the USB cable from one of the drives without taking it offline:

[root@chef /mnt/Chef]# zpool status farcryz1
  pool: farcryz1
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    farcryz1    ONLINE       0     0     0
      raidz1    ONLINE       0     0     0
        da4     ONLINE      22     4     0
        da3     ONLINE       0     0     0
        da1     ONLINE       0     0     0
        da2     ONLINE       0     0     0

errors: No known data errors

Data's still there, pool still online. Great! Now let's try to restore the pool. I plugged the drive back in, and issued the zpool replace command as I was instructed to above:

[root@chef /mnt/Chef]# zpool replace farcryz1 da4
invalid vdev specification
use '-f' to override the following errors:
/dev/da4 is part of active pool 'farcryz1'

Um.... That's not helpful... So I tried a zpool clear farcryz1, but that didn't help at all. I still couldn't replace da4. So I tried a combination of onlineing, offlineing, clearing, replaceing, and scrubing. Now I am stuck here:

[root@chef /mnt/Chef]# zpool status -v farcryz1
  pool: farcryz1
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
    invalid.  Sufficient replicas exist for the pool to continue
    functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-4J
 scrub: scrub completed after 0h2m with 0 errors on Fri Sep  9 13:43:34 2011
config:

    NAME        STATE     READ WRITE CKSUM
    farcryz1    DEGRADED     0     0     0
      raidz1    DEGRADED     0     0     0
        da4     UNAVAIL      9     0     0  experienced I/O failures
        da3     ONLINE       0     0     0
        da1     ONLINE       0     0     0
        da2     ONLINE       0     0     0

errors: No known data errors
[root@chef /mnt/Chef]# zpool replace farcryz1 da4
cannot replace da4 with da4: da4 is busy

How can I recover from this situation, where one device in my zpool was unexpectedly disconnected (but is not a failed device) and is now back again, ready to be resilvered?


EDIT: As requested, a tail of dmesg:

(ses3:umass-sim4:4:0:1): removing device entry
(da4:umass-sim4:4:0:0): removing device entry
ugen3.2: <Western Digital> at usbus3
umass4: <Western Digital My Book 1140, class 0/0, rev 3.00/10.03, addr 1> on usbus3
da4 at umass-sim4 bus 4 scbus6 target 0 lun 0
da4: <WD My Book 1140 1003> Fixed Direct Access SCSI-6 device 
da4: 400.000MB/s transfers
da4: 1907697MB (3906963456 512 byte sectors: 255H 63S/T 243197C)
ses3 at umass-sim4 bus 4 scbus6 target 0 lun 1
ses3: <WD SES Device 1003> Fixed Enclosure Services SCSI-6 device 
ses3: 400.000MB/s transfers
ses3: SCSI-3 SES Device
GEOM: da4: partition 1 does not start on a track boundary.
GEOM: da4: partition 1 does not end on a track boundary.
GEOM: da4: partition 1 does not start on a track boundary.
GEOM: da4: partition 1 does not end on a track boundary.
ugen3.2: <Western Digital> at usbus3 (disconnected)
umass4: at uhub3, port 1, addr 1 (disconnected)
(da4:umass-sim4:4:0:0): lost device
(da4:umass-sim4:4:0:0): removing device entry
(ses3:umass-sim4:4:0:1): lost device
(ses3:umass-sim4:4:0:1): removing device entry
ugen3.2: <Western Digital> at usbus3
umass4: <Western Digital My Book 1140, class 0/0, rev 3.00/10.03, addr 1> on usbus3
da4 at umass-sim4 bus 4 scbus6 target 0 lun 0
da4: <WD My Book 1140 1003> Fixed Direct Access SCSI-6 device 
da4: 400.000MB/s transfers
da4: 1907697MB (3906963456 512 byte sectors: 255H 63S/T 243197C)
ses3 at umass-sim4 bus 4 scbus6 target 0 lun 1
ses3: <WD SES Device 1003> Fixed Enclosure Services SCSI-6 device 
ses3: 400.000MB/s transfers
ses3: SCSI-3 SES Device
share|improve this question

2 Answers 2

up vote 4 down vote accepted

Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'.

Looks like after the initial temporary failure, you may only have needed to do a zpool clear to clear the errors.

If you want to pretend that it's a drive replacement, you probably need to clear the data off the drive first before you try re-adding it to the pool.

share|improve this answer
    
"you may only have needed to do a zpool clear to clear the errors" -- so the extra commands I ran probably caused the situation I'm in now. I suspected that as well. "you probably need to clear the data off the drive first before you try re-adding it to the pool" -- So, dd if=/dev/zero of=/dev/da4 bs=1M? –  Josh Sep 9 '11 at 21:13
    
Sure, blow it away, pretend its a brand-new drive. –  MikeyB Sep 9 '11 at 21:15
    
Thanks; doing so now. –  Josh Sep 9 '11 at 21:21
    
This did it, thanks! Before this would work, I had to reboot, and after doing so, zpool replace farcryz1 da4 responded with cannot replace da4 with da4: no such pool or dataset. But hooking up another USB drive as da4 and then the newly-zeroed 2TB drive after as da5 allowed me to zpool replace farcryz1 da4 da5. Thanks! –  Josh Sep 10 '11 at 13:46

What were the outputs of the various commands you tried? Did you try the -f switch on any of them?

Did you run zpool clear poolname device-name?

In your case, zpool clear farcryz1 da4 - That should have gotten the resilvering process underway.

share|improve this answer
    
I tried zpool clear farcryz1 da4, and it produces no output and no change at all. I just did that and I am now seeing da4 UNAVAIL 3 0 0 experienced I/O failures. –  Josh Sep 9 '11 at 20:34
    
And can you reboot? What does a tail of dmesg say? –  ewwhite Sep 9 '11 at 20:42
    
I can reboot, but have not yet. I will try that, but, should that be necessary? I have added a tail of dmesg but it's not very useful... –  Josh Sep 9 '11 at 20:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.