Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

NOTE: My understanding of this question has changed significantly since I first asked it (see Edit 2 below) but I've left the original version intact.

We've put together an offsite backup system (still testing in-house) that does its data transfer via ZFS send/receive. Machines on both ends are FreeBSD 8.2. Overall, the setup works well.

However, there's clearly something I don't understand about ZFS snapshot stream sizes. I've had a hard time finding information on this, so I'm hoping someone with some more experience can enlighten me.

On the source machine, I have a filesystem of about 47GB for which I need to transfer snapshots:

# zfs list -t snapshot -r -s creation stg/serverx
stg/serverx@20110620  2.88M      -  47.1G  -
stg/serverx@20110621  2.89M      -  47.1G  -
stg/serverx@20110622  2.88M      -  47.1G  -
stg/serverx@20110623  5.44M      -  46.6G  -

I have the snapshot from 6/22 already on the remote server, so I send it the stream generated by

zfs send -i stg/serverx@20110622 stg/serverx@20110623

This is received on the other end with no trouble; however, the stream generated is over 80 gigabytes—nearly twice the size of the entire source filesystem.

Am I misinterpreting the meaning of the "USED" column generated by zfs list? I'd have expected this snapshot stream to be 5.44M plus a certain amount of overhead. It seems I don't quite understand what constitutes overhead.

Possibly useful information: We backup (via rsync) each server to its own filesystem. This particular one seems to generate the largest streams (relative to filesystem and snapshot size). I suspect it may be related to the fact that it's a mail server, so some of its content is pretty dynamic. However, I'd expect this to show up in the snapshot "used" size too.

Obviously, we can save quite a bit by compressing the stream (it can probably be reduced to 12-20% of its original size). Even then, bandwidth will be our limiting factor, so we'd like to understand what makes these streams so large and whether we can do anything to mitigate it.

EDIT: I had forgotten that we have zfs compression enabled on the source filesystem. So the 47-ish gigabytes used does indeed translate to nearly 80 gigabytes of "real" filesystem data. I suppose that's a partial explanation, but I still don't see why the incremental stream from zfs send would be so large.


Further investigation of this backup and a couple of others has led to the conclusion that the large transfers were in fact to be expected (due to some upgrades that had taken place). However, I don't see any indication of that large amount of data in the output from zfs list.

I've been through documentation, and I understand that there are a lot of complexities in computing the space used by a snapshot. The zfs man page says the following in the description of the used property:

When snapshots . . . are created, their space is initially shared between the snapshot and the file system, and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot, and counted in the snapshot's space used.

This makes sense to me. However, I would expect to see a much larger snapshot created at the end of the day the server was upgraded. In fact, it's only a few megabytes. There is no deduplication here (zpool version 15). However, the incremental stream generated by zfs send -i is quite large, and contains all the upgrade information.

Can someone explain this apparent inconsistency? The related question, then, is: How can I get a reasonable estimate of the size of an incremental stream (from, for example, zfs list output)?

share|improve this question

What type of email system and what type of "store" technology? If the mail store is already compressed in any way, then each incremental may actually be a full as its compressed data stream may be dynamically changing due to its compression.

Also is dedup in play across either system? It sounds like there may be a remote chance that it might be on the source system. That might account for the size difference.

share|improve this answer
No deduplication--this is zpool version 15 on both sides. +1 for suggesting compression-related issues; that turns out to be a partial explanation. See also my edit 2 above. (Sorry to be slow in follow-up here.) – eaj Jun 28 '11 at 14:28
What is the actual data format being backed up from your email server? File-based individual message files (Unix-style), Exchange EDB, backup format from a backup application??? It seems like the "zfs send" may not be moving the data in "native zfs," but possibly expanding it back to its original format and then letting the receiving system "recode" things back into "zfs" storage/format. – user48838 Jun 28 '11 at 20:21
It's not just the mail being backed up, it's the entire server. Anyway, the mail is in plain old mbox format. The zfs send command is surely removing the zfs compression when it creates the stream, but that size difference isn't near enough to account for what I'm seing. – eaj Jun 28 '11 at 21:33
OK, so what is the ratio (text vs. binary vs. compressed stream) of the data being backed up? The zfs compression ratios are directly impacted by the data it is working with - example: text is more compressible than binary, where an already compresses zip file may no longer be compressed any further. – user48838 Jun 28 '11 at 22:02

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.