16

Using a striped mirror (Raid 10) as an example, if two disks are expanded to four, how can the data from the two disks be 're-balanced' (spread out) amongst the two mirrors?

That is, how can the files which were written to one mirror be written to two mirrors to take advantage of the striped disks?

2
  • 9
    @ewwhite Sometimes you don't have that option.
    – Moshe Katz
    Jul 6, 2017 at 19:45
  • 7
    @ewwhite And sometimes growth cannot be anticipated correctly, because it depends on the outside market. Your advice would only be valid if you also had a working time machine.
    – user121391
    Jul 7, 2017 at 7:12

3 Answers 3

12

Only newly written (appended) data is split between all currently active vdevs, there is no explicit rebalance operation in ZFS.

Two possible solutions:

  1. Wait until all old data is written again (because of CoW, this can take a very long time, in the worst case double the time it would take to write the disk completely).
  2. Remove all data and write it anew (zfs send/recv is helpful in getting all data off the pool and back without losing anything). This does not have to be done all in one go, and it can be done on the same pool.

To be more precise, I would choose the second solution and transfer each file system separately at times where system load is low (for example at night):

  • Take a snapshot (zfs snapshot -r) of a decently sized file system (and decendant file systems, recursively)
  • Use zfs send | zfs recv with appropriate options to send the snapshot to a newly created temporary file system (can be on the same pool if space permits); this file system should be on the same location in the hierarchy as the old one
  • After the copy is done (may take some time, because the disks have to read and write), zfs destroy the old snapshot and old file system
  • zfs rename the temporary system to the old name
  • Check and change mount points with zfs mount, rearranging the previous situation for your replaced file system
  • Repeat until all file systems are moved
1
  • 2
    If this would be done to same pool, wouldn't it affect fragmentation?
    – gsl
    Mar 5, 2021 at 13:41
7

possible 3rd solution (as mentioned by SirMaster in this FreeNAS forum post) :

  • add new disk(s) to zpool
  • copy many files to another new /mnt/pool/temp/ directory
  • delete original files: rm -rf original/
  • rename dir back: mv temp/ original/

This works because ZFS will proportionally place writes to whichever vdev has most free space, in this case the brand new drives which were empty. (as of 0.7, zfs will favor faster drives for writes, but lets assume your 2 new drives are same or greater performance than the original drives)

It's probably slower than zfs send | zfs recv, but simpler because you don't have to create/destroy snapshots.

You can run zpool list -v before and after to see each vdev utilization.

Also, found a php script that does the copy/delete/rename procedure on a file-by-file basis. This was linked in an answer from a similar (but zvol) question a few years older. (didn't test this script personally)

3
  • 3
    Thanks, I used this technique to rebalance my home lab after expanding the pool with a new raidz1 vdev. Worked a trick. Aug 9, 2020 at 2:53
  • Be aware - this approach may not work as you would expect...see Cyberjock's answers down in the thread linked in this post.
    – quantum
    Mar 10, 2022 at 14:42
  • 1
    @quantum That "cyberjock" dude from a decade ago seemed to be making stuff up. He could never give never answer any specific questions about IOPS after rebalancing beyond claiming he's some sort of expert.
    – Navin
    Mar 13, 2022 at 8:45
0

This is similar to user121391's answer, but I just completed a rebalancing operation using Jim Salter's excellent blog post from a few years ago. Essentially, you:

  1. Scrub your pool and check the status.
  2. Free up a disk from each mirror using zpool detach.
  3. Create a new pool using the freed disks.
  4. Copy the data to the new pool. I used his syncoid for this.
  5. Scrub the new pool and verify the status.
  6. zpool destroy your old pool after muttering whatever good luck charms you need to mutter.
  7. zpool attach the remaining disks to the new pool.
  8. Rename the new pool to the original name.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .