I've been tasked with making an off-site backup work over the WAN. Both storage boxes are FreeBSD based NAS boxes running ZFS.

Once or twice a week, 15-60 gigs of photography data gets dumped to the office NAS. My job is to figure out how to get this data off-site as reliably as possible using the VERY SLOW DSL connection (~700Kb/s upload). The receiving box is in much better shape, at 30Mb/s down, 5Mb/s up.

I know, carrying a hard drive off-site would move data much more quickly, but it's not an option in this case.

My options seem to be either:

  • ZFS incremental send over ssh
  • Rsync

rsync is a time honored solution, and has the all-important ability to resume a send if something gets interrupted. It has the disadvantage of iterating over many files and not knowing about dedup.

ZFS snapshot sending might transfer a bit less data (it knows a lot more about the file system, can do dedup, can package up the metadata changes more efficiently than rsync) and has the advantage of properly duplicating the filesystem state, rather than simply copying files individually (which is more disk intensive).

I'm concerned about ZFS replication performance[1] (though that article is a year old). I'm also concerned about being able to re-start the transfer if something goes down - the snapshot capability doesn't seem to include that. The whole system needs to be completely hands-off.

[1] http://wikitech-static.wikimedia.org/articles/z/f/s/Zfs_replication.html

Using either option, I should be able to de-prioritize the traffic by routing it through a specified port, then using the QOS on the routers. I need to avoid a major negative impact on users at both sites during each transfer, since it will take several days.

So... that's my thinking on the issue. Have I missed any good options? Has anyone else set something similar up?

  1. If you can transfer a maximum of 6GB per day (assuming zero overhead and zero competing traffic) and you are needing to move "15-60 gigs" at a frequency of "once or twice per week," that works out to 15-120 GB per week, or anywhere from 2-17 GB per day. Because it is necessary to plan for peak demand, and 17 GB is far in excess of even your theoretical maximum of 6 GB, it is likely that you have a very serious bandwidth problem. What will it take to upgrade the connection? If upgrading the connection is impossible, please consider the option of mailing physical media on a scheduled basis (e.g. weekly).

  2. Assuming that you can get the bandwidth math to make a little bit more sense, rsync is likely to be the best option. Deduplication awareness would be hugely valuable when replicating highly redundant data (e.g. virtual machine images), but it should have little or no benefit when it comes to unique digital content (audio, video, photos)... unless, of course, users are inadvertently storing duplicate copies of identical files.

  • I figure I can use the available bandwidth, and most of the data dumps tend towards the smaller end of the range. Practically, it's gonna be around 2-3 gigs a day average, judging from a past month of data. I don't need the replication immediately. – Paul McMillan Oct 10 '10 at 2:12
  • And yeah, mailing physical media is far better... I wish it were an option. – Paul McMillan Oct 10 '10 at 2:12
  • Good point about dedup. Most of what gets copied won't be duplicated - the users aren't quite that dense. – Paul McMillan Oct 10 '10 at 2:13
  • 1
    The only thing I would add is maybe not using rsync. I too experienced the slowness of rsync because I was using it as a transfer process, not sync process. Then I realized most of my existing data didn't change and only new data needed to be copied, for me, I used cp on only the new files and it was much faster. If I had files that changed (or only parts of files) then I would use rsync. So I suggest separating the new files out and pick a resumable transfer method. Also, compression would be a CPU & RAM/bandwidth trade-off (on both ends). – Scott McClenning Oct 10 '10 at 3:30
  • Hmm... I've read that with proper configuration, rsync can be made to go relatively quickly. How much optimization did you attempt? – Paul McMillan Oct 10 '10 at 5:48

After doing some research I believe you are right about sending snapshots. The ZFS SEND and RECEIVE commands can be piped into bzip2 and then that file can be rsync-ed to the other machine.

Here are some sources I used:

I hadn't found any posts with replication scripts posted, but I did find someone that posted their backup script. That said, I didn't understand it so it may be junk.

Many of the website talked about setting up a cron job to do this frequently. If this is the case, you could replicate/backup with less impact to bandwidth and users and be a good disaster recovery feature because the offsite data is more up to date. (That is, after the initial chunk of data when getting started.)

Again, I think you had the right idea sending snapshots there seems to be a lot of advantages to using SEND / RECEIVE.

EDIT: Just watched a video1 video2 that may helps suports the use of SEND/RECEIVE and talks about rsync (starts at 3m49s). Ben Rockwood was the speaker and here is a link to his blog.

  • 1
    I guess the use of rsync there is limited to the pause/resume functionality, rather than the actual file diffing. This makes sense, since the file system itself (and the change files it generates) knows better than rsync what's going on. – Paul McMillan Oct 12 '10 at 19:06
  • As an additional note: ZSTD, a modern faster replacement for gzip and bzip, supports multiple threads, and more than 20 compression levels. It also has a contributed optional feature called 'adaptive compression'. With this mode, the compression level is automatically tuned up and down as needed to keep the network pipe full, while doing as much compression as possible to save time. This prevents you from doing so much compression that it becomes a bottleneck, or missing out on compression you could be doing because the network is too slow. – Allan Jude Sep 6 '18 at 0:37

What is the purpose of the backups and how will they need to be accessed?

If your backups are mainly for disaster recovery then ZFS snapshots might be preferable as you'll be able to get a filesystem back to the exact state it was in at the time of the last incremental.

However, if your backups are also supposed to provide users access to files that might have been accidentally deleted, corrupted, etc. then rsync could be a better option. End users may not understand the concept of snapshots or perhaps your NAS doesn't provide end users access to previous snapshots. In either case you can use rsync to provide a backup that is easily accessible to the user via the filesystem.

With rsync you can use the --backup flag to preserve backups of files that have been changed, and with the --suffix flag you can control how old versions of files are renamed. This makes it easy to create a backup where you might have dated old versions of files like


You can easily combine this with a cronjob containing a find command to purge any old files as needed.

Both solutions should be able to preserve enough metainformation about files to work as a backup (rsync provides --perms, --owner etc. flags). I use rsync to backup large amounts of data between datacenters and am very happy with the setup.


ZFS should receive the 'resumable send' feature, which will allow continuing an interrupted replication some time around March of this year. The feature has been completed by Matt Ahrens and some other people, and should be upstreamed soon.

  • Just a note that 'resumable send' has been in OpenZFS (on FreeBSD, Linux, MacOS, etc) for quite some time now. There is also a 'compressed send' feature now, where data will stay compressed as it is on disk, as part of the replication stream. – Allan Jude Sep 6 '18 at 0:35

Maybe WAN compression device would a solution...? we use Riverbed and we are quite happy with them (eg NetApp SnapMirror is being very well compressed, up to 80-90%)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.