4

I'm running Ubuntu 14.04 with ZOL version v0.6.5.4:

root@box ~# dmesg | egrep "SPL|ZFS"
[   34.430404] SPL: Loaded module v0.6.5.4-1~trusty
[   34.475743] ZFS: Loaded module v0.6.5.4-1~trusty, ZFS pool version 5000, ZFS filesystem version 5

ZFS is configured in raidz2 across 6x 2TB Seagate SpinPoint M9T 2.5" drives, with a read cache, deduplication and compression enabled:

root@box ~# zpool status -v
  pool: bigpool
 state: ONLINE
config:

        NAME                                           STATE     READ WRITE CKSUM
        bigpool                                        ONLINE       0     0     0
          raidz2-0                                     ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S37<redactedid> ONLINE       0     0     0
            ata-ST2000LM003_HN-M201RAD_S34<redactedid> ONLINE       0     0     0
        cache
          sda3                                         ONLINE       0     0     0

Every few days, the box will lock up, and I'll get errors such as:

blk_update_request: I/O Error, dev sdh, sector 764218200
blk_update_request: I/O Error, dev sdf, sector 764218200
blk_update_request: I/O Error, dev sde, sector 764218200
blk_update_request: I/O Error, dev sdd, sector 764218200
blk_update_request: I/O Error, dev sdc, sector 764218432
blk_update_request: I/O Error, dev sdg, sector 764218200

smartctl shows that the disks are not recording any SMART errors, and they're all fairly new disks. I find it odd too that they're all failing on the same sector (with the exception of sdc). I was able to grab a screenshot of the terminal (I can't ssh in once the errors start):

console errors

Perhaps this is a controller failing, or a bug related to zfs?

1
  • Were you able to track down the cause of these errors? I'm in a similar situation, and I'm having a hard time figuring out what the underlying issue is.
    – Dominic P
    Jul 13, 2016 at 19:56

1 Answer 1

0

You have a controller, cabling or backplane problem. Note how all drives are impacted at the same time...

I'd also caution against using deduplication on a setup like this unless it's totally necessary.

4
  • Thank you, but how did you arrive at this being a controller/cabling/backplane problem? Is it just the timing of the failure that indicates this? Is there something I can do to confirm that this is indeed the issue, and not something software based (causing all drives to fail at the same time)?
    – dymk
    Feb 20, 2016 at 22:15
  • 2
    ZFS doesn't fail like that. I doubt you have a software issue. It's far more likely to be your hardware: specifically the controller or whatever is between your disks and the controller.
    – ewwhite
    Feb 20, 2016 at 22:17
  • I've swapped out the backplane, RAID controller, cabling, and motherboard (the parts from the previous server are the PSUs, RAM, CPUs, and disks), and I'm still getting the same errors: i.imgur.com/EzFU3D3.png Could this possibly be a CPU or RAM issue?
    – dymk
    Mar 19, 2016 at 17:15
  • You have ext4 errors too. So this is probably still a hardware problem.
    – ewwhite
    Mar 19, 2016 at 17:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .