I've been having trouble backing up a ZFS filesytem due to my backup devices running out of space.

The first time I forgot that compression was enabled. However, for the second attempt, I got the compression ratio of the file system, and also got the apparent size as returned by du.

These agreed with each other on the size:

louis@watson:~$ sudo zpool list
watson  3.62T  2.74T   904G    75%  1.00x  ONLINE  -

louis@watson:~$ sudo zfs list 
watson         2.74T   846G    30K  none
watson/gelato  2.73T   846G  2.67T  /data/gelato

louis@watson:~$ sudo zfs get compressratio watson/gelato
watson/gelato  compressratio  1.64x  -

I expected 2.67 TB compressed with a savings of 1.64x to fit on a 6 TB drive, as reaffirmed by du:

louis@watson:~$ cd /data/gelato/
louis@watson:/data/gelato$ sudo du -hs --apparent-size
4.6T    .

But I once again ran out of space during the backup and I'm not sure why. There was probably less than 1 TB left to copy when the 6 TB drive ran out of space. Still this is way over anything I can think would account for the discrepancy.


louis@watson:~$ sudo zfs list -o space
watson          846G  2.74T         0     30K              0      2.74T
watson/gelato   846G  2.73T     53.6G   2.67T              0          0
watson/home     846G  15.8G     57.3M   15.8G              0          0

Is there a better way to get the actual size of the data stored on a compressed ZFS filesystem?

I should mention I use rsync -avh to backup. On the destination drive I don't see the snapshots(...../.zfs/snapshot). Does that mean the snapshots aren't being copied?


Some things to consider:

  • Check the record size (with zfs list -o recsize watson/gelato) of both source and target. If you have lots of small data but a large record size, space is wasted. The other way round, space is also wasted because of headers and metadata, but usually the effect is not as noticeable. If you share the file system via SMB/CIFS, you can look at the difference using the Windows Explorer folder properties window.
  • Check sector alignment (ashift) on both drives and compare with the drive specifications (can be found in the technical data sheets of the drive model). Wrong aligment can negatively affect space of your pool (in this example he lost about 9%).
  • Check if the copies property has ever been set to a value of 1 or greater (this may have been set and disabled in the past, and copies would have been created for any newly written data in the period between).
  • Get more details about how the space is used with the properties usedbychildren, usedbydataset, usedbyrefreservation, and usedbysnapshots. They sum up to used, so it will be nothing new, but could help to indentify old snapshots and the like.
  • To see the amount of space used by data and metadata as if compression was deactivated, check the properties logicalused and logicalreferenced
  • Because of differences in specifying data sizes (base 2 vs. base 10) your 6 TB drive actually has only about 5.457 TiB (9% less than assumed).
  • I'm actually just backing up to a plain old external drive (about 5.5 TB formatted). – Louis Nov 11 '17 at 1:10
  • The record size is 128KB. Would the target respect/duplicate this? I assume you are pointing out a 1 byte file will be 128 KB, on the ZFS FS. Or that any <128 KB file will wind up using 128 KB of storage. Is that right? – Louis Nov 11 '17 at 6:27
  • I inhereted this guy, who will actually break with a dist-upgrade, so am stuck in 2012, if that matters. I don't see those props (logical*). The used by props agree with other reporting: i.imgur.com/odNcutm.png – Louis Nov 11 '17 at 6:32
  • 1
    @Louis 128 KB is the default, so if you don't change it, the second system will have it too. It will only be changed if you do it manually, or if create a descendant file system and inheritance rules apply. Yes, the small sizes taking up full 128KB is what I meant. – user121391 Nov 13 '17 at 8:20
  • I think it came down to file size vs size on disk, which is what I guess you were steering me at with record size. Still not sure how one can estimate this cost beforehand – Louis Nov 18 '17 at 1:35

I'd suggest zpool list.

I'd also check snapshot usage. Are there any snapshots in your environment?

  • yes, thanks. I checked the size of the snapshots and they're very small. – Louis Nov 11 '17 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.