1

I am trying to rsync data one of the volumes in a ZFS pool to an exfat formatted drive. The volume is only 1.3 TB but the rsync got up to syncing 3+ TB worth of data. I killed the rsync so I could figure out what is wrong.

The ZFS pool:

ubuntu@ubuntu:~$ zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
backup  7.25T  3.59T  3.66T         -     0%    49%  1.00x  ONLINE  -

The volumes and their mountpoints:

ubuntu@ubuntu:~$ zfs list
NAME                                                      USED  AVAIL  REFER  MOUNTPOINT
backup                                                   2.61T  2.49T  35.4M  /backup
backup/.system                                           1.97M  2.49T   140K  legacy
backup/.system/configs-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx   593K  2.49T   593K  legacy
backup/.system/cores                                      692K  2.49T   692K  legacy
backup/.system/rrd-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx       128K  2.49T   128K  legacy
backup/.system/samba4                                     337K  2.49T   337K  legacy
backup/.system/syslog-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx    128K  2.49T   128K  legacy
backup/vol1                                              1.26T  2.49T  1.26T  /backup/vol1
backup/vol2                                               128K  2.49T   128K  /backup/vol2
backup/vol3                                              1.78G  2.49T  1.78G  /backup/vol3
backup/vol4                                              1.34T  2.49T  1.34T  /backup/vol4
backup/vol5                                              4.51G  2.49T  4.51G  /backup/vol5

The size on disk of /backup/vol1:

ubuntu@ubuntu:~$ du -sh /backup/vol1
1.3T    /backup/vol1

How much data was copied after doing an rsync for a couple days:

ubuntu@ubuntu:~$ rsync -avzh --progress --no-o --no-g /backup/vol1 /media/ubuntu/external_drive/freenas/
...
...
ubuntu@ubuntu:~$ du -sh /media/ubuntu/external_drive/freenas/vol1
3.2T    /media/ubuntu/external_drive/freenas/vol1

I killed it after seeing how much was transferred.

2
  • 1
    Do you have compression enabled on ZFS? That could explain the discrepancy. To query the uncompressed size, use “zfs list -o logicalused”. I’m not sure how rsync treats empty (zeroed) blocks, so it could also be counting those even though ZFS isn’t.
    – Dan
    Nov 6, 2018 at 5:34
  • Even logicalused only shows 1.4 TB. Nov 6, 2018 at 20:10

1 Answer 1

0

Just to expand on my comment above, this will happen under two circumstances that I can think of:

  • The files / volumes you're sending are using block saving techniques (compression, dedup, or snapshots/clones). ZFS has saved you some space internally, so the ZFS property used is small, but rsync still has to read all the logical (uncompressed, undeduped, un-snapshot-sharing) data because it has no idea about those features. You can figure out how much logical data there is by running zfs list -o logicalused (or logicalreferenced if you have snapshots).
  • The files / volumes you're trying to send are sparse. From your comment above, I believe this is the case you're in. rsync is sending all the unwritten blocks of your volume, even though they're all zeroed out (because they've never been written to). Try using the --sparse argument to rsync, as described in this answer, to work around this behavior, and then using --in-place for subsequent syncs to avoid re-syncing the entire file instead of just the parts that have changed.
2
  • @IMTheNachoMan How did it go? Don’t forget to accept / upvote the answer if it helped :).
    – Dan
    Nov 17, 2018 at 11:09
  • I am still working on it. The computer I was copying to had issues and now I am on vacation. I will continue when I get back from vacation and report back then. Nov 23, 2018 at 21:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .