4

I have a relatively new box here running Ubuntu with ZFS-on-Linux across an 18-disk array. The first couple of times I did a zpool scrub, it seemed like everything worked nicely, but the last several times I've tried, it completely trashes the system - when I can get it to do anything at all, it's reporting a load average of 20-50, and most of the time it's just unresponsive.

This isn't expected behavior is it? Is there some configuration I can change to make this less awful?

Update:

  • Hardware configuration is basically http://www.45drives.com/products/storage/s45-lite.php
  • 18 6TB disks connected via a HighPoint R750 HBA.
  • Ubuntu 14.04 LTS, Linux 3.19 kernel.
  • Pool is currently around 50% full.
  • No deduplication.
  • [Update 2]: disks are arranged in 2 RAIDZ (single-parity) vdevs with 9 disks each.
3
  • What's your device configuration? Hopefully not one vdev that has all 18 drives in a RAIDZ2 pool - that will be a horrible array to do scans/scrubs on no matter how well you tune things. See solarisinternals.com/wiki/index.php/… Apr 2, 2016 at 18:23
  • Hmm, 9 disks each... that's still a bad design.
    – ewwhite
    Apr 3, 2016 at 17:16
  • @ewwhite "bad" because of concerns about data integrity or some other reason? If the former, we've done our analysis and decided it's acceptable for our current needs. If the latter, then can you elaborate?
    – arcticmac
    Apr 4, 2016 at 20:04

2 Answers 2

3

Ended up asking on the ZFS-on-Linux mailing list, and eventually figured out that my problem was that the ZFS ARC was using too much of the system RAM (or more specifically, not leaving enough free). This was resulting in contention over memory between other Kernel tasks and ZFS, with the result that there was a lot of I/O on the system (non-ZFS) disk, and things slowed to a crawl. Interestingly, the system did not report any swap usage and all of the paging going on was reading static pages from disk (presumably libraries and the like), which is why my usual "it's swapping so it will be slow" alarm didn't go off.

I decreased the size of the ARC cache to leave 3-4GB of RAM free, and now things are running much better!

0

You have to tune your ZFS installation. The defaults on ZoL aren't geared towards your use case.

Please provide details on your hardware setup and ZFS configuration.

  • Is your pool over 80% full?
  • Do you have deduplication enabled?
  • What types of disks and controllers are you using?

In general, you can modify the following in the zfs.conf module config file.

options zfs zfs_vdev_scrub_min_active=48
options zfs zfs_vdev_scrub_max_active=128
2
  • updated original post with some hardware setup info. Will try out your suggestions once I'm able to stop the scrub...
    – arcticmac
    Apr 1, 2016 at 1:52
  • Looks like the current scrub max/min settings are 2 and 1 respectively. I'm not clear how those parameters work exactly, but it seems like increasing the number of concurrent operations isn't likely to make it use fewer threads?
    – arcticmac
    Apr 1, 2016 at 22:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .