7

NOTE: My understanding of this question has changed significantly since I first asked it (see Edit 2 below) but I've left the original version intact.

We've put together an offsite backup system (still testing in-house) that does its data transfer via ZFS send/receive. Machines on both ends are FreeBSD 8.2. Overall, the setup works well.

However, there's clearly something I don't understand about ZFS snapshot stream sizes. I've had a hard time finding information on this, so I'm hoping someone with some more experience can enlighten me.

On the source machine, I have a filesystem of about 47GB for which I need to transfer snapshots:

# zfs list -t snapshot -r -s creation stg/serverx
NAME                   USED  AVAIL  REFER  MOUNTPOINT
(.......)
stg/serverx@20110620  2.88M      -  47.1G  -
stg/serverx@20110621  2.89M      -  47.1G  -
stg/serverx@20110622  2.88M      -  47.1G  -
stg/serverx@20110623  5.44M      -  46.6G  -

I have the snapshot from 6/22 already on the remote server, so I send it the stream generated by

zfs send -i stg/serverx@20110622 stg/serverx@20110623

This is received on the other end with no trouble; however, the stream generated is over 80 gigabytes—nearly twice the size of the entire source filesystem.

Am I misinterpreting the meaning of the "USED" column generated by zfs list? I'd have expected this snapshot stream to be 5.44M plus a certain amount of overhead. It seems I don't quite understand what constitutes overhead.

Possibly useful information: We backup (via rsync) each server to its own filesystem. This particular one seems to generate the largest streams (relative to filesystem and snapshot size). I suspect it may be related to the fact that it's a mail server, so some of its content is pretty dynamic. However, I'd expect this to show up in the snapshot "used" size too.

Obviously, we can save quite a bit by compressing the stream (it can probably be reduced to 12-20% of its original size). Even then, bandwidth will be our limiting factor, so we'd like to understand what makes these streams so large and whether we can do anything to mitigate it.

EDIT: I had forgotten that we have zfs compression enabled on the source filesystem. So the 47-ish gigabytes used does indeed translate to nearly 80 gigabytes of "real" filesystem data. I suppose that's a partial explanation, but I still don't see why the incremental stream from zfs send would be so large.


EDIT 2:

Further investigation of this backup and a couple of others has led to the conclusion that the large transfers were in fact to be expected (due to some upgrades that had taken place). However, I don't see any indication of that large amount of data in the output from zfs list.

I've been through documentation, and I understand that there are a lot of complexities in computing the space used by a snapshot. The zfs man page says the following in the description of the used property:

When snapshots . . . are created, their space is initially shared between the snapshot and the file system, and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot, and counted in the snapshot's space used.

This makes sense to me. However, I would expect to see a much larger snapshot created at the end of the day the server was upgraded. In fact, it's only a few megabytes. There is no deduplication here (zpool version 15). However, the incremental stream generated by zfs send -i is quite large, and contains all the upgrade information.

Can someone explain this apparent inconsistency? The related question, then, is: How can I get a reasonable estimate of the size of an incremental stream (from, for example, zfs list output)?

2 Answers 2

6

I know this is a really old question, but I've seen it a few difference places. There's always been some confusion about the value expressed in zfs list as it pertains to using zfs send|recv. The problem is that the value expressed in the USED column is actually an estimate of the amount of space that will be released if that single snapshot is deleted, bearing in mind that there may be earlier and later snapshots referencing the same data blocks.

Example:

zfs list -t snapshot -r montreve/cev-prod | grep 02-21
NAME                                      USED  AVAIL  REFER  MOUNTPOINT
montreve/cev-prod@2018-02-21_00-00-01     878K      -   514G  -
montreve/cev-prod@2018-02-21_sc-daily     907K      -   514G  -
montreve/cev-prod@2018-02-21_01-00-01    96.3M      -   514G  -
montreve/cev-prod@2018-02-21_02-00-01    78.5M      -   514G  -
montreve/cev-prod@2018-02-21_03-00-01    80.3M      -   514G  -
montreve/cev-prod@2018-02-21_04-00-01    84.0M      -   514G  -
montreve/cev-prod@2018-02-21_05-00-01    84.2M      -   514G  -
montreve/cev-prod@2018-02-21_06-00-01    86.7M      -   514G  -
montreve/cev-prod@2018-02-21_07-00-01    94.3M      -   514G  -
montreve/cev-prod@2018-02-21_08-00-01     101M      -   514G  -
montreve/cev-prod@2018-02-21_09-00-01     124M      -   514G  -

In order find out how much data will need to be transferred to reconstitute a snapshot via zfs send|recv, you'll need to use the dry-run feature (-n) for these values. Taking the above-listed snapshots try:

zfs send -nv -I montreve/cev-prod@2018-02-21_00-00-01 montreve/cev-prod@2018-02-21_09-00-01
send from @2018-02-21_00-00-01 to montreve/cev-prod@2018-02-21_sc-daily estimated size is 1.99M
send from @2018-02-21_sc-daily to montreve/cev-prod@2018-02-21_01-00-01 estimated size is 624M
send from @2018-02-21_01-00-01 to montreve/cev-prod@2018-02-21_02-00-01 estimated size is 662M
send from @2018-02-21_02-00-01 to montreve/cev-prod@2018-02-21_03-00-01 estimated size is 860M
send from @2018-02-21_03-00-01 to montreve/cev-prod@2018-02-21_04-00-01 estimated size is 615M
send from @2018-02-21_04-00-01 to montreve/cev-prod@2018-02-21_05-00-01 estimated size is 821M
send from @2018-02-21_05-00-01 to montreve/cev-prod@2018-02-21_06-00-01 estimated size is 515M
send from @2018-02-21_06-00-01 to montreve/cev-prod@2018-02-21_07-00-01 estimated size is 755M
send from @2018-02-21_07-00-01 to montreve/cev-prod@2018-02-21_08-00-01 estimated size is 567M
send from @2018-02-21_08-00-01 to montreve/cev-prod@2018-02-21_09-00-01 estimated size is 687M
total estimated size is 5.96G

Yikes! That's a whole heck of a lot more than the USED values. However, if you don't need all of the intermediary snapshots at the destination, you can use the consolidate option (-i rather than -I), which will calculate the necessary differential between any two snapshots even if there others in between.

zfs send -nv -i montreve/cev-prod@2018-02-21_00-00-01 montreve/cev-prod@2018-02-21_09-00-01
send from @2018-02-21_00-00-01 to montreve/cev-prod@2018-02-21_09-00-01 estimated size is 3.29G
total estimated size is 3.29G

So that's isolating the various blocks that were rewritten between snapshots, so we only take their final state.

But that's not the whole story! zfs send is based on extracting the logical data from the source, so that if you have compression activated on the source filesystem, the estimates are based on the uncompressed data that will need to be sent. For example, taking one incremental snapshot and writing it to disk you get something close to the estimated value from the dry-run command:

zfs send -i montreve/cev-prod@2018-02-21_08-00-01 montreve/cev-prod@2018-02-21_09-00-01 > /montreve/temp/cp08-09.snap
-rw-r--r--  1 root root    682M Feb 22 10:07 cp08-09.snap

But if you pass it through gzip, we see that the data is significantly compressed:

zfs send -i montreve/cev-prod@2018-02-21_08-00-01 montreve/cev-prod@2018-02-21_09-00-01 | gzip > /montreve/temp/cp08-09.gz
-rw-r--r--  1 root root    201M Feb 22 10:08 cp08-09.gz

Side note - this is based on the OpenZFS on Linux, version : - ZFS: Loaded module v0.6.5.6-0ubuntu16

You will find some references to optimisations that can be applied to the send stream (-D deduplicated stream, -e more compact), but with this version I haven't observed any impact on the size of the streams generated with my datasets.

2
  • Massive kudos for such a detailed response to an ancient question! I've long since left that job and that system, but I believe you've managed to answer the question. Thanks!
    – eaj
    Feb 26, 2018 at 14:54
  • Hopefully it will be useful to others as well. It's non-obvious until you've had to dive into the details like I had to recently. :-) Feb 26, 2018 at 15:08
1

What type of email system and what type of "store" technology? If the mail store is already compressed in any way, then each incremental may actually be a full as its compressed data stream may be dynamically changing due to its compression.

Also is dedup in play across either system? It sounds like there may be a remote chance that it might be on the source system. That might account for the size difference.

4
  • No deduplication--this is zpool version 15 on both sides. +1 for suggesting compression-related issues; that turns out to be a partial explanation. See also my edit 2 above. (Sorry to be slow in follow-up here.)
    – eaj
    Jun 28, 2011 at 14:28
  • What is the actual data format being backed up from your email server? File-based individual message files (Unix-style), Exchange EDB, backup format from a backup application??? It seems like the "zfs send" may not be moving the data in "native zfs," but possibly expanding it back to its original format and then letting the receiving system "recode" things back into "zfs" storage/format.
    – user48838
    Jun 28, 2011 at 20:21
  • It's not just the mail being backed up, it's the entire server. Anyway, the mail is in plain old mbox format. The zfs send command is surely removing the zfs compression when it creates the stream, but that size difference isn't near enough to account for what I'm seing.
    – eaj
    Jun 28, 2011 at 21:33
  • OK, so what is the ratio (text vs. binary vs. compressed stream) of the data being backed up? The zfs compression ratios are directly impacted by the data it is working with - example: text is more compressible than binary, where an already compresses zip file may no longer be compressed any further.
    – user48838
    Jun 28, 2011 at 22:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .