4

I'm running Ubuntu 14.04 with ZOL version v0.6.5.4:

root@box ~# dmesg | egrep "SPL|ZFS"
[   34.430404] SPL: Loaded module v0.6.5.4-1~trusty
[   34.475743] ZFS: Loaded module v0.6.5.4-1~trusty, ZFS pool version 5000, ZFS filesystem version 5

ZFS is configured in raidz2 across 6x 2TB Seagate SpinPoint M9T 2.5" drives, with a read cache, deduplication and compression enabled:

root@box ~# zpool status -v
  pool: bigpool
 state: ONLINE
config:

        NAME                                           STATE     READ WRITE CKSUM
        bigpool                                        ONLINE       0     0     0
          raidz2-0                                     ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S34<redactedid> ONLINE       0     0     0
        cache
          sda3                                         ONLINE       0     0     0

Every few days, the box will lock up, and I'll get errors such as:

blk_update_request: I/O Error, dev sdh, sector 764218200
blk_update_request: I/O Error, dev sdf, sector 764218200
blk_update_request: I/O Error, dev sde, sector 764218200
blk_update_request: I/O Error, dev sdd, sector 764218200
blk_update_request: I/O Error, dev sdc, sector 764218432
blk_update_request: I/O Error, dev sdg, sector 764218200

smartctl shows that the disks are not recording any SMART errors, and they're all fairly new disks. I find it odd too that they're all failing on the same sector (with the exception of sdc). I was able to grab a screenshot of the terminal (I can't ssh in once the errors start):

console errors

Perhaps this is a controller failing, or a bug related to zfs?

  • Were you able to track down the cause of these errors? I'm in a similar situation, and I'm having a hard time figuring out what the underlying issue is. – Dominic P Jul 13 '16 at 19:56
0

You have a controller, cabling or backplane problem. Note how all drives are impacted at the same time...

I'd also caution against using deduplication on a setup like this unless it's totally necessary.

  • Thank you, but how did you arrive at this being a controller/cabling/backplane problem? Is it just the timing of the failure that indicates this? Is there something I can do to confirm that this is indeed the issue, and not something software based (causing all drives to fail at the same time)? – dymk Feb 20 '16 at 22:15
  • 1
    ZFS doesn't fail like that. I doubt you have a software issue. It's far more likely to be your hardware: specifically the controller or whatever is between your disks and the controller. – ewwhite Feb 20 '16 at 22:17
  • I've swapped out the backplane, RAID controller, cabling, and motherboard (the parts from the previous server are the PSUs, RAM, CPUs, and disks), and I'm still getting the same errors: i.imgur.com/EzFU3D3.png Could this possibly be a CPU or RAM issue? – dymk Mar 19 '16 at 17:15
  • You have ext4 errors too. So this is probably still a hardware problem. – ewwhite Mar 19 '16 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.