4

While installing new hardware today, I involuntarily disconnected the power to one of our servers. This server is running FreeBSD 10 with Root-on-ZFS as offered by the installer. It has a total of 36 disks, distributed into two RAID-Z2 which belong to the same pool (called zroot).

After switching the server back on, it will now no longer boot. Before the boot menu shows up the following messages are printed:

Loading /boot/defaults/loader.conf ZFS: i/o error - all block copies unavailable Warning: error reading file /boot/loader.conf

Despite these messages, the system continues to boot the kernel until it stops at a mountfrom> prompt and I can't get it to continue from there. If I enter zfs:zroot/ROOT/default, it just says unknown filesystem.

I can however boot from a USB stick, import the zpool and I can read both /boot/defaults/loader.conf and /boot/loader.conf. In fact, it appears that the zpool is perfectly fine.

I have then tried to reinstall the bootcode:

gpart bootcode -b /tmp/zroot/boot/pmbr -p /boot/gptzfsboot -i 1 da0

I have tried to recreate the zpool cache file:

zpool set cachefile=/tmp/zroot/boot/zfs/zpool.cache zroot

I have tried to manually tell the bootloader to load the ZFS module.

All of these attempts still resulted in the same behaviour and I am now completely stuck on what I could possibly do to make the system boot again.

Edit:

Output of zpool list:

$ zpool list
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  97.8T  28.0T  69.7T    28%  1.33x  ONLINE  /tmp/zfs

Currently the system is booted from a USB stick, thus I could not import the pool to /

Contents of /boot/loader.conf:

$ cat /boot/loader.conf
zfs_load="YES"
ipmi_load="YES"

Edit 2:

$ zpool status
  pool: zroot
 state: ONLINE
  scan:
config:

    NAME                                            STATE     READ WRITE CKSUM
    zroot                                           ONLINE       0     0     0
      raidz2-0                                      ONLINE       0     0     0
        gptid/f8c57b3a-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/f97b7e8b-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fa3c41d9-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/faf62101-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fbb19e1b-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fc6b75db-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fd26cd36-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fddb4b8e-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fe9a55f6-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/ff582110-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/001713d1-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/00d90b6c-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/0192be91-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/023ea058-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/02fb8ee4-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/03ab78ec-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/04632542-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/052144fd-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
      raidz2-1                                      ONLINE       0     0     0
        da18                                        ONLINE       0     0     0
        da19                                        ONLINE       0     0     0
        da20                                        ONLINE       0     0     0
        da21                                        ONLINE       0     0     0
        da22                                        ONLINE       0     0     0
        da23                                        ONLINE       0     0     0
        da24                                        ONLINE       0     0     0
        da25                                        ONLINE       0     0     0
        da26                                        ONLINE       0     0     0
        da27                                        ONLINE       0     0     0
        da28                                        ONLINE       0     0     0
        da29                                        ONLINE       0     0     0
        da30                                        ONLINE       0     0     0
        da31                                        ONLINE       0     0     0
        da32                                        ONLINE       0     0     0
        da33                                        ONLINE       0     0     0
        da34                                        ONLINE       0     0     0
        da35                                        ONLINE       0     0     0

errors: No known data errors
Improve this question
5
  • (good warning NOT to use a ZFS root)
    – ewwhite
    Jul 31, 2014 at 17:12
  • Could you provide the output of /boot/loader.conf and zpool list?
    – b13n1u
    Jul 31, 2014 at 17:32
  • Perhaps your pool is too large for BIOS to handle and you've written something needed for boot after the reachable area? How big is the pool? What type of pool is it?
    – Steve Wills
    Jul 31, 2014 at 19:03
  • It's certainly possible that the BIOS can't handle the pool size, as you can see it is a very large pool. But then why did the system boot before?
    – Sarek
    Aug 1, 2014 at 8:09
  • Those are very large zpools. The ZFS docs recommend that zpools be a maximum of 10 drives per pool. You can combine pools to get a larger array.
    – Stefan Lasiewski
    Nov 15, 2014 at 23:09

2 Answers 2

Reset to default
4

Ok, never mind what I said in my other answer. You won't be able to boot into single-user mode if you are having problems with ZFS partitions anyway. What's funny is after I posted my answer, I had a problem with power again and I couldn't boot into my system, just like your problem.

So here is what I did to get back in.

I used a FreeBSD 10.0-RELEASE Live CD (it was a bootonly I think -- to minimize download)

  1. Boot into the CD and choose "Live CD" from the options. (log in as root)

  2. Execute the following commands to attach partitions using geli; you would have to enter your passphrase if you have a root-on-ZFS full disk encryption setup

     mkdir /tmp/bootpool
 zpool import -f bootpool
 zfs set mountpoint=/tmp/bootpool bootpool
 zfs mount -a
 cp /tmp/bootpool/boot/encryption.key /tmp/
 zfs umount -a
 zfs set mountpoint=/bootpool bootpool
 zpool export bootpool
 geli attach -k /tmp/encryption.key /dev/ada0p4
 zpool import -f -R /mnt zroot

I had to zpool import with -f flag because the power down problem didn't export the partition in the right way.

After executing these commands I rebooted and the system went online again.

Though, I had another problem with bootpool not loading correctly after boot.

The /boot folder was empty, it was also a symbolic link to the /bootpool/boot directory. I also found out that when importing the zfs bootpool partition it created /bootpool/boot/boot so the issue was the 2 boot folders under /bootpool.

So I did this after reboot to load the bootpool manually.

      zpool export bootpool
      zpool import -f bootpool
      (cd to root and remove the old symbolic link `boot`)
      cd /
      rm boot
      (notice the double `boot` directory issue)
      ln -sf bootpool/boot/boot/

That's it and the /boot symbolic link worked again and I could load the missing kernel modules, for example kldload linux or anything else.

Hope that helps, and if anybody knows a clean fix for the bootpool problem above please share it with me.

Thanks,

Draco

References:

Improve this answer
2

Have you tried booting into single-user mode and force mount ZFS partitions?

# zfs mount -a

I've experienced a power failure on my FreeBSD 10 (which is also Root-on-ZFS) and I managed to get it to boot using the command above.

Let me know how it goes.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.