NOTE: My understanding of this question has changed significantly since I first asked it (see Edit 2 below) but I've left the original version intact.
We've put together an offsite backup system (still testing in-house) that does its data transfer via ZFS send/receive. Machines on both ends are FreeBSD 8.2. Overall, the setup works well.
However, there's clearly something I don't understand about ZFS snapshot stream sizes. I've had a hard time finding information on this, so I'm hoping someone with some more experience can enlighten me.
On the source machine, I have a filesystem of about 47GB for which I need to transfer snapshots:
# zfs list -t snapshot -r -s creation stg/serverx NAME USED AVAIL REFER MOUNTPOINT (.......) stg/serverx@20110620 2.88M - 47.1G - stg/serverx@20110621 2.89M - 47.1G - stg/serverx@20110622 2.88M - 47.1G - stg/serverx@20110623 5.44M - 46.6G -
I have the snapshot from 6/22 already on the remote server, so I send it the stream generated by
zfs send -i stg/serverx@20110622 stg/serverx@20110623
This is received on the other end with no trouble; however, the stream generated is over 80 gigabytes—nearly twice the size of the entire source filesystem.
Am I misinterpreting the meaning of the "USED" column generated by
zfs list? I'd have expected this snapshot stream to be 5.44M plus a certain amount of overhead. It seems I don't quite understand what constitutes overhead.
Possibly useful information: We backup (via rsync) each server to its own filesystem. This particular one seems to generate the largest streams (relative to filesystem and snapshot size). I suspect it may be related to the fact that it's a mail server, so some of its content is pretty dynamic. However, I'd expect this to show up in the snapshot "used" size too.
Obviously, we can save quite a bit by compressing the stream (it can probably be reduced to 12-20% of its original size). Even then, bandwidth will be our limiting factor, so we'd like to understand what makes these streams so large and whether we can do anything to mitigate it.
EDIT: I had forgotten that we have zfs compression enabled on the source filesystem. So the 47-ish gigabytes used does indeed translate to nearly 80 gigabytes of "real" filesystem data. I suppose that's a partial explanation, but I still don't see why the incremental stream from
zfs send would be so large.
Further investigation of this backup and a couple of others has led to the conclusion that the large transfers were in fact to be expected (due to some upgrades that had taken place). However, I don't see any indication of that large amount of data in the output from
I've been through documentation, and I understand that there are a lot of complexities in computing the space used by a snapshot. The
zfs man page says the following in the description of the
When snapshots . . . are created, their space is initially shared between the snapshot and the file system, and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot, and counted in the snapshot's space used.
This makes sense to me. However, I would expect to see a much larger snapshot created at the end of the day the server was upgraded. In fact, it's only a few megabytes. There is no deduplication here (zpool version 15). However, the incremental stream generated by
zfs send -i is quite large, and contains all the upgrade information.
Can someone explain this apparent inconsistency? The related question, then, is: How can I get a reasonable estimate of the size of an incremental stream (from, for example,
zfs list output)?