Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a home server using FreeBSD and ZFS that has worked well for the last 5 years, and on several occasions I have successfully replace faulty disks.

However, today a minor disaster has happened, and I'm hoping to find a solution.

I have a top-level pool that consists of 3 vdevs, each of which is a raidz1 pool, so up to 3 disks can fail -- assuming that they all belong to different vdevs -- and data integrity is intact.

Yesterday, I noticed quite a few errors being reported by 1 disk in 1 vdev. From past experience, this usually indicates that the disk is about to fail, so I do what I usually do:

  1. Offline the disk: zpool offline tank gpt/ta4
  2. Physically replace the disk
  3. Set up the new disk with gpart, and then zpool replace tank gpt/ta4

However, this time between steps 2 and 3 disaster struck: when I powered on the server after installing the new drive, I smelled something burning, and my HBA indicated that 4 of the drives weren't available! By a stroke of unbelievably bad luck, there must have been some voltage surge, because another drive in the same vdev (gpt/ta2) is now completely dead, and visual inspection reveals one of the MOSFETs on the PCB is blown.

So now gpt/ta2 is UNAVAIL and gpt/ta4 is OFFLINE, so obviously the vdev, which is raidz1, is also UNAVAIL.

My questions are: 1) Is there a way to bring gpt/ta4 back online? When I try to issue "zpool online tank gpt/ta4", it tells me that the pool is unavailable, so I can't do so. I can understand why this may be so, but I was thinking that gpt/ta4, although experiencing some read errors, was basically still a 'good' member of the raidz1 pool prior to taking it offline (zpool status reported there were no known data errors). Is there anyway to achieve this?

2) Failing that, is there a way to at least bring the remainder of my top-level pool (which consists of 3 raidz1 vdevs) online? The other 2 vdevs are perfectly fine.

Please help, I have a lot of precious data on it :-)

Thanks in advance.

share|improve this question
add comment

1 Answer

up vote 5 down vote accepted

Not that it helps you at this juncture, but this is precisely why you'll never see me advising people use raidz1 - and for mirror sets, if they're using huge disks, often suggesting triple-mirrors.

It is /unlikely in the extreme/ that any act you can take is going to get tank back online. I must start with that, so as not to raise your hopes.

1: Make sure the disks are safe - even if that means unplugging all of them.

2: Update to the latest version of FreeBSD - you want the latest ZFS bits you can get your hands on.

3: Put the original gpt/ta4 (that is supposedly 'OK' and just experiencing read errors) back in the system or into a new system with newer ZFS bits (as well as all the others if you've removed them), boot it, and run, in order until one works (be forewarned - these are not safe, especially the last one, in that in their attempts to recover the system they're likely to roll back and thus lose recently written data):

  • zpool import -f tank
  • zpool import -fF tank
  • zpool import -fFX tank

If all 3 fail, you're outside the realm of "simple" recovery. Some Googling for 'importing bad pools', 'zdb', 'zpool import -F', 'zpool import -X', 'zpool import -T' (danger!), and the like, might provide you some additional blogs and information on recovery attempts made by others, but it's already on very dangerous and potentially further-data-damaging ground at that point and you're rapidly entering territory of paid recovery services (and not from traditional data recovery companies, they have zero expertise with ZFS and will not be of any use to you).

Note: A more precise and 'safer' method would be to 'zpool import -o readonly=on -f -T [txg_id] tank'. However, for this to work, you'd need to use zdb on your own, first, to locate a seemingly healthy recent txg_id, and I'm not prepared to try to explain all that here. Google will be your friend here - take no action until you've read sufficient information to feel somewhat comfortable with what you're doing. Trust no single source.

Note 2: the 'safest' thing to do would be to immediately contact someone capable of ZFS recovery services.

Note 3: the next 'safest' thing to do would be to put the drives in a safe system and dd each entire raw drive to a new disk, as well, giving you, theoretically, identical copies of your disks, but that would mean you'd need a like number of new disks, preferably of similar or identical size/type to the old ones but not strictly necessary. And only then attempt any of the above on one set of the drives while keeping the others aside for safe-keeping.

share|improve this answer
    
Thank you so much for your very detailed and clear response. I will definitely be doing my homework now on the various importing methods, zdb, txg_id, etc. Anything I can salvage would be a bonus at this point, as I have reconciled myself to the very likely outcome of not being able to recover my data. The disks are currently safe: I have shutdown the machine and it will remain in that state until I am ready to take further action. –  indotoonster Dec 22 '13 at 22:50
1  
Also, another course of action that someone else has suggested is to swap PCBs (ta2 and ta4 are identical), but I thought it would be best to exhaust all possible software-based solutions before getting my hands dirty. –  indotoonster Dec 22 '13 at 22:55
2  
Hi Nex7, once again I'd like to thank you for pointing me in the direction of the many robust importing options of zpool, I've read up a lot on the matter. In any case, though, I have wonderful news to report: I ended up biting the bullet and swapping the PCBs given that the drives are identical, and lo and behold, /gpt/ta2 is back. I have now replaced /gpt/ta4 and it is resilvering as I type. This has served as a very valuable lesson for me! –  indotoonster Dec 23 '13 at 15:24
    
Very glad you got it to work out, @indotoonster! This is an area that ZFS really needs both more documentation about and an easier/more robust set of options/tools to deal with weird situations. Hopefully developers will donate their time and effort at streamlining this sort of stuff, and it will become better understood as time goes on. :) –  Nex7 Dec 27 '13 at 22:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.